Freedman Justin Pettit Jianying Luo and Nick McKeown Stanford University Scott Shenker UC Berkeley and ICSI ABSTRACT This paper presents Ethane a new network architecture for the enterprise Ethane allows managers to de64257ne a single network wide 6 ID: 84832
Download Pdf The PPT/PDF document "Ethane Taking Control of the Enterprise ..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
frequently,andtheyareeasilyspoofed.Theloosebindingbetweenusersandtheirtrafcisaconstanttargetforattacksinenterprisenetworks.Ifthenetworkistobegovernedbyapolicydeclaredoverhigh-levelnames(e.g.,usersandhosts)thenpacketsshouldbeidentiable,withoutdoubt,ascomingfromaparticularphysicalentity.Thisrequiresastrongbindingbetweenauser,themachinetheyareusing,andtheaddressesinthepacketstheygenerate.Thisbindingmustbekeptconsistentatalltimes,bytrackingusersandmachinesastheymove.Toachievetheseaims,wefollowedtheleadofthe4Dproject[14]andadoptedacentralizedcontrolarchitecture.Centralizedsolu-tionsarenormallyananathemafornetworkingresearchers,butwefeelitistheproperapproachforenterprisemanagement.IP'sbest-effortservicemodelisbothsimpleandunchanging,well-suitedfordistributedalgorithms.Networkmanagementisquitetheopposite;itsrequirementsarecomplexandrequirestrongconsistency,mak-ingitquitehardtocomputeinadistributedmanner.Therearemanystandardobjectionstocentralizedapproaches,suchasresilienceandscalability.However,aswediscusslaterinthepaper,ourresultssuggestthatstandardreplicationtechniquescanprovideexcellentresilience,andcurrentCPUspeedsmakeitpossibletomanageallcontrolfunctionsonasizablenetwork(e.g.,25,000hosts)fromasinglecommodityPC.EthanebearssubstantialresemblancetoSANE,ourrecently-proposedclean-slateapproachtoenterprisesecurity[12].SANEwas,asaremanyclean-slatedesigns,difculttodeployandlargelyuntested.WhileSANEcontainedmanyvaluableinsights,Ethaneextendsthispreviousworkinthreemainways:Securityfollowsmanagement.Enterprisesecurityis,inmanyways,asubsetofnetworkmanagement.Bothrequireanetworkpolicy,theabilitytocontrolconnectivity,andthemeanstoobservenetworktrafc.Networkmanagementwantsthesefeaturessoastocontrolandisolateresources,andthentodiagnoseandxerrors,whereasnetworksecurityseekstocontrolwhoisallowedtotalktowhom,andthentocatchbadbehaviorbeforeitpropagates.WhendesigningEthane,wedecidedthatabroadapproachtonetworkmanagementwouldalsoworkwellfornetworksecurity.Incrementaldeployability.SANErequireda“fork-lift"replace-mentofanenterprise'sentirenetworkinginfrastructureandchangestoalltheend-hosts.Whilethismightbesuitableinsomecases,itisclearlyasignicantimpedimenttowidespreadadoption.Ethaneisdesignedsothatitcanbeincrementallydeployedwithinanen-terprise:itdoesnotrequireanyhostmodications,andEthaneSwitchescanbeincrementallydeployedalongsideexistingEther-netswitches.Signicantdeploymentexperience.Ethanehasbeenimplementedinbothsoftwareandhardware(special-purposeGigabitEthernetswitches)anddeployedatStanford'sComputerSciencedepartmentforoverfourmonthsandmanagedover300hosts.Thisdeploymentexperiencehasgivenusinsightintotheoperationalissuessuchadesignmustconfront,andresultedinsignicantchangesandex-tensionstotheoriginaldesign.Inthispaper,wedescribeourexperiencesdesigning,implement-ing,anddeployingEthane.Webeginwithahigh-leveloverviewoftheEthanedesignin§2,followedbyadetaileddescriptionin§3.In§4,wedescribeapolicylanguagePol-EththatwebuilttomanageourEthaneimplementation.Wethendiscussourimplementationanddeploymentexperience(§5),followedbyperformanceanaly-sis(§6).Finallywepresentlimitations(§7),discussrelatedwork(§8),andthenconclude(§9). 2.OVERVIEWOFETHANEDESIGNEthanecontrolsthenetworkbynotallowinganycommunica-tionbetweenend-hostswithoutexplicitpermission.Itimposesthisrequirementthroughtwomaincomponents.TherstisacentralControllercontainingtheglobalnetworkpolicythatdeterminesthefateofallpackets.WhenapacketarrivesattheController—howitdoessoisdescribedbelow—theControllerdecideswhethertheowrepresentedbythatpacket1shouldbeallowed.TheControllerknowstheglobalnetworktopologyandperformsroutecomputa-tionforpermittedows.Itgrantsaccessbyexplicitlyenablingowswithinthenetworkswitchesalongthechosenroute.TheControllercanbereplicatedforredundancyandperformance.ThesecondcomponentisasetofEthaneSwitches.Incon-trasttotheomniscientController,theseSwitchesaresimpleanddumb.ConsistingofasimpleowtableandasecurechanneltotheController,Switchessimplyforwardpacketsunderthedirec-tionoftheController.Whenapacketarrivesthatisnotintheowtable,theyforwardthatpackettotheController(inamannerwedescribelater),alongwithinformationaboutwhichportthepacketarrivedon.Whenapacketarrivesthatisintheowtable,itisfor-wardedaccordingtotheController'sdirective.NoteveryswitchinanEthanenetworkneedstobeanEthaneSwitch:OurdesignallowsSwitchestobeaddedgradually,andthenetworkbecomesmoremanageablewitheachadditionalSwitch.2.1Names,Bindings,andPolicyLanguageWhentheControllerchecksapacketagainsttheglobalpolicy,itisevaluatingthepacketagainstasetofsimplerules,suchas“GuestscancommunicateusingHTTP,butonlyviaawebproxy”or“VoIPphonesarenotallowedtocommunicatewithlaptops.”Ifwewanttheglobalpolicytobespeciedintermsofsuchphysicalentities,weneedtoreliablyandsecurelyassociateapacketwiththeuser,group,ormachinethatsentit.Ifthemappingsbetweenma-chinenamesandIPaddresses(DNS)orbetweenIPaddressesandMACaddresses(ARPandDHCP)arehandledelsewhereandareunauthenticated,thenwecannotpossiblytellwhosentthepacket,eveniftheuserauthenticateswiththenetwork.Thisisanotoriousandwidespreadweaknessincurrentnetworks.With(logical)centralization,itissimpletokeepthenamespaceconsistentascomponentsjoin,leaveandmovearoundthenetwork.NetworkstatechangessimplyrequireupdatingthebindingsattheController.Thisisincontrasttotoday'snetworkwheretherearenowidelyusedprotocolsforkeepingthisinformationconsistent.Fur-ther,distributingthenamespaceamongallswitcheswouldgreatlyincreasethetrustedcomputingbaseandrequirehighoverheadstomaintainconsistencyoneachbindevent.InEthane,wealsouseasequenceoftechniquestosecurethebindingsbetweenpacketheadersandthephysicalentitiesthatsentthem.First,Ethanetakesoverallthebindingofaddresses.WhenmachinesuseDHCPtorequestanIPaddress,Ethaneassignsitknowingtowhichswitchportthemachineisconnected,enablingEthanetoattributeanarrivingpackettoaphysicalport.2Second,thepacketmustcomefromamachinethatisregisteredonthenet-work,thusattributingittoaparticularmachine.Finally,usersarerequiredtoauthenticatethemselveswiththenetwork—forexam- 1AllpoliciesconsideredinEthanearebasedoverows,wheretheheadereldsusedtodeneaowarebasedonthepackettype(forexample,TCP/UDPowsincludetheEthernet,IPandtransportheaders).Thus,onlyasinglepolicydecisionneedbemadeforeachsuch“ow”.2Aswediscusslater,aprimaryadvantageofknowingtheingressportofapacketisthatitallowstheControllertoapplylterstotherst-hopswitchusedbyunwantedtrafc. WhenweaddanEthaneSwitchtothenetwork,ithastondtheController(§3.3),openasecurechanneltoit,andhelptheCon-trollergureoutthetopology.Wedothiswithamodiedmini-mumspanningtreealgorithm(per§3.7anddenotedbythick,solidlinesinthegure).TheoutcomeisthattheControllerknowsthewholetopology,whileeachSwitchonlyknowsapartofit.Whenweadd(orboot)ahost,ithastoauthenticateitselfwiththeController.FromtheSwitch'spoint-of-view,packetsfromthenewhostaresimplypartofanewow,andsopacketsareautomaticallyforwardedtotheControlleroverthesecurechannel,alongwiththeIDoftheSwitchportonwhichtheyarrived.TheControllerauthenticatesthehostandallocatesitsIPaddress(theControllerincludesaDHCPserver).3.2SwitchesAwiredEthaneSwitchislikeasimpliedEthernetswitch.IthasseveralEthernetinterfacesthatsendandreceivestandardEth-ernetpackets.Internally,however,theswitchismuchsimpler,asthereareseveralthingsthatconventionalEthernetswitchesdothatanEthaneswitchdoesn'tneed:AnEthaneSwitchdoesn'tneedtolearnaddresses,supportVLANs,checkforsource-addressspoof-ing,orkeepow-levelstatistics(e.g.,startandendtimeofows,althoughitwilltypicallymaintainper-owpacketandbytecoun-tersforeachowentry).IftheEthaneSwitchisreplacingaLayer-3“switch”orrouter,itdoesn'tneedtomaintainforwardingtables,ACLs,orNAT.Itdoesn'tneedtorunroutingprotocolssuchasOSPF,ISIS,andRIP.NordoesitneedseparatesupportforSPANsandport-replication(thisishandleddirectlybytheowtableunderthedirectionoftheController).Itisalsoworthnotingthattheowtablecanbeseveralorders-of-magnitudesmallerthantheforwardingtableinanequivalentEth-ernetswitch.InanEthernetswitch,thetableissizedtominimizebroadcasttrafc:asswitchesoodduringlearning,thiscanswamplinksandmakesthenetworklesssecure.5Asaresult,anEthernetswitchneedstorememberalltheaddressesit'slikelytoencounter;evensmallwiringclosetswitchestypicallycontainamillionen-tries.EthaneSwitches,ontheotherhand,canhavemuchsmallerowtables:theyonlyneedtokeeptrackofowsin-progress.Forawiringcloset,thisislikelytobeafewhundredentriesatatime,smallenoughtobeheldinatinyfractionofaswitchingchip.Evenforacampus-levelswitch,whereperhapstensofthousandsofowscouldbeongoing,itcanstilluseon-chipmemorythatsavescostandpower.WeexpectanEthaneSwitchtobefarsimplerthanitscorre-spondingEthernetswitch,withoutanylossoffunctionality.Infact,weexpectthatalargeboxofpower-hungryandexpensiveequip-mentwillbereplacedbyahandfulofchipsonaboard.FlowTableandFlowEntries.TheSwitchdatapathisaman-agedowtable.FlowentriescontainaHeader(tomatchpacketsagainst),anAction(totelltheswitchwhattodowiththepacket),andPer-FlowData(whichwedescribebelow).Therearetwocommontypesofentryintheowtable:per-owentriesdescribingapplicationowsthatshouldbeforwarded,andper-hostentriesthatdescribemisbehavinghostswhosepacketsshouldbedropped.ForTCP/UDPows,theHeadereldcoverstheTCP/UDP,IP,andEthernetheaders,aswellasphysicalportinformation.TheassociatedActionistoforwardthepackettoaparticularinterface,updateapacket-and-bytecounter(inthePer-FlowData),andsetanactivitybit(sothatinactiveentriescanbetimed-out).Formisbehavinghosts,theHeadereldcontainsan 5Infact,networkadministratorsoftenusemanuallyconguredandinexibleVLANstoreduceooding. Ethernetsourceaddressandthephysicalingressport.6Theassoci-atedActionistodropthepacket,updateapacket-and-bytecounter,andsetanactivitybit(totellwhenthehosthasstoppedsending).OnlytheControllercanaddentriestotheowtable.Entriesareremovedbecausetheytimeoutduetoinactivity(localdecision)orbecausetheyarerevokedbytheController.TheControllermightrevokeasingle,badlybehavedow,oritmightremoveawholegroupofowsbelongingtoamisbehavinghost,ahostthathasjustleftthenetwork,orahostwhoseprivilegeshavejustchanged.Theowtableisimplementedusingtwoexact-matchtables:Oneforapplication-owentriesandoneformisbehaving-hosten-tries.Becauseowentriesareexactmatches,ratherthanlongest-prexmatches,itiseasytousehashingschemesinconventionalmemoriesratherthanexpensive,power-hungryTCAMs.OtherActionsarepossibleinadditiontojustforwardanddrop.Forexample,aSwitchmightmaintainmultiplequeuesfordiffer-entclassesoftrafc,andtheControllercantellittoqueuepack-etsfromapplicationowsinaparticularqueuebyinsertingqueueIDsintotheowtable.Thiscanbeusedforend-to-endL2iso-lationforclassesofusersorhosts.ASwitchcouldalsoperformaddresstranslationbyreplacingpacketheaders.Thiscouldbeusedtoobfuscateaddressesinthenetworkby“swapping”addressesateachSwitchalongthepath—aneavesdropperwouldnotbeabletotellwhichend-hostsarecommunicating—ortoimplementad-dresstranslationforNATinordertoconserveaddresses.Finally,aSwitchcouldcontroltherateofaow.TheSwitchalsomaintainsahandfulofimplementation-specicentriestoreducetheamountoftrafcsenttotheController.ThisnumbershouldremainsmalltokeeptheSwitchsimple,althoughthisisatthediscretionofthedesigner.Ononehand,suchentriescanreducetheamountoftrafcsenttotheController;ontheotherhand,anytrafcthatmissesontheowtablewillbesenttotheControlleranyway,sothisisjustanoptimization.LocalSwitchManager.TheSwitchneedsasmalllocalmanagertoestablishandmaintainthesecurechanneltotheController,tomonitorlinkstatus,andtoprovideaninterfaceforanyadditionalSwitch-specicmanagementanddiagnostics.(WeimplementedourmanagerintheSwitch'ssoftwarelayer.)TherearetwowaysaSwitchcantalktotheController.Therstone,whichwehaveassumedsofar,isforSwitchesthatarepartofthesamephysicalnetworkastheController.Weexpectthistobethemostcommoncase;e.g.,inanenterprisenetworkonasinglecampus.Inthiscase,theSwitchndstheControllerusingourmodiedMinimumSpanningTreeprotocoldescribedin§3.7.TheprocessresultsinasecurechannelstretchingthroughtheseintermediateSwitchesallthewaytotheController.IftheSwitchisnotwithinthesamebroadcastdomainastheController,theSwitchcancreateanIPtunneltoit(afterbeingmanuallyconguredwithitsIPaddress).ThisapproachcanbeusedtocontrolSwitchesinarbitrarylocations,e.g.,theothersideofaconventionalrouterorinaremotelocation.Inoneapplica-tionofEthane,theSwitch(mostlikelyawirelessaccesspoint)isplacedinahomeorsmallbusinessandthenmanagedremotelybytheControlleroverthissecuretunnel.ThelocalSwitchmanagerrelayslinkstatustotheControllersoitcanreconstructthetopologyforroutecomputation.Switchesmaintainalistofneighboringswitchesbybroadcastingandreceiv-ingneighbor-discoverymessages.NeighborlistsaresenttotheControllerafterauthentication,onanydetectablechangeinlinkstatus,andperiodicallyevery15seconds. 6Ifahostisspoong,itsrst-hopportcanbeshutofdirectly(§3.3). SwitchhasbeenconguredwiththeController'scredentialsandtheControllerwiththeSwitches'credentials.IfaSwitchndsashorterpathtotheController,itattemptstwo-wayauthenticationwithitbeforeadvertisingthatpathasavalidroute.Therefore,theminimumspanningtreegrowsradiallyfromtheController,hop-by-hopaseachSwitchauthenticates.Authenticationisdoneusingthepreconguredcredentialstoen-surethatamisbehavingnodecannotmasqueradeastheControlleroranotherSwitch.Ifauthenticationissuccessful,theSwitchcre-atesanencryptedconnectionwiththeControllerthatisusedforallcommunicationbetweenthepair.Bydesign,theControllerknowstheupstreamSwitchandphys-icalporttowhicheachauthenticatingSwitchisattached.AfteraSwitchauthenticatesandestablishesasecurechanneltotheCon-troller,itforwardsallpacketsitreceivesforwhichitdoesnothaveaowentrytotheController,annotatedwiththeingressport.ThisincludesthetrafcofauthenticatingSwitches.Therefore,theControllercanpinpointtheattachmentpointtothespanningtreeofallnon-authenticatedSwitchesandhosts.OnceaSwitchauthenticates,theControllerwillestablishaowinthenetworkbetweenitselfandtheSwitchforthesecurechannel.4.THEPOL-ETHPOLICYLANGUAGEPol-EthisalanguagefordeclaringpolicyinanEthanenetwork.WhileEthanedoesn'tmandateaparticularlanguage,wedescribePol-Ethasanexample,toillustratewhat'spossible.Wehaveim-plementedPol-Ethanduseitinourprototypenetwork.4.1OverviewInPol-Eth,networkpolicyisdeclaredasasetofrules,eachcon-sistingofaconditionandacorrespondingaction.Forexample,theruletospecifythatuserbobisallowedtocommunicatewiththewebserver(usingHTTP)isthefollowing:[(usrc="bob")^(protocol="http")^(hdst="websrv")]:allow;Conditions.Conditionsareaconjunctionofzeroormorepred-icateswhichspecifythepropertiesaowmusthaveinorderfortheactiontobeapplied.Fromtheprecedingexamplerule,iftheuserinitiatingtheowis“bob”andtheowprotocolis“HTTP”andtheowdestinationishost“websrv,”thentheowisallowed.Thelefthandsideofapredicatespeciesthedomain,andtherighthandsidegivestheentitiestowhichitapplies.Forexample,thepredicate(usrc=“bob”)appliestoallowsinwhichthesourceisuserbob.Validdomainsinclude{usrc,udst,hsrc,hdst,apsrc,apdst,protocol},whichrespectivelysignifytheuser,host,andac-cesspointsourcesanddestinationsandtheprotocoloftheow.InPol-Eth,thevaluesofpredicatesmayincludesinglenames(e.g.,“bob”),listofnames(e.g.,[“bob”,“linda”]),orgroupinclu-sion(e.g.,in(“workstations”)).AllnamesmustberegisteredwiththeControllerordeclaredasgroupsinthepolicyle,asdescribedbelow.Actions.Actionsincludeallow,deny,waypoints,andoutbound-only(forNAT-likesecurity).Waypointdeclarationsincludealistofentitiestoroutetheowthrough,e.g.,waypoints(“ids”,“web-proxy”).4.2RuleandActionPrecedencePol-Ethrulesareindependentanddon'tcontainanintrinsicor-dering;thus,multipleruleswithconictingactionsmaybesatis-edbythesameow.Conictsareresolvedbyassigningprioritiesbasedondeclarationorder.Ifoneruleprecedesanotherinthepol-icyle,itisassignedahigherpriority. #Groups— desktops=["grifn","roo"]; laptops=["glaptop","rlaptop"]; phones=["gphone","rphone"]; server=["http_server","nfs_server"]; private=["desktops","laptops"]; computers=["private","server"]; students=["bob","bill","pete"]; profs=["plum"]; group=["students","profs"]; waps=["wap1","wap2"]; %% #Rules— [(hsrc=in("server")^(hdst=in("private"))]:deny; #Donotallowphonesandprivatecomputerstocommunicate [(hsrc=in("phones")^(hdst=in("computers"))]:deny; [(hsrc=in("computers")^(hdst=in("phones"))]:deny; #NAT-likeprotectionforlaptops [(hsrc=in("laptops")]:outbound-only; #Norestrictionsondesktopscommunicatingwitheachother [(hsrc=in("desktops")^(hdst=in("desktops"))]:allow; #Forwireless,non-groupmemberscanusehttpthrough #aproxy.Groupmembershaveunrestrictedaccess. [(apsrc=in("waps"))^(user=in("group"))]:allow; [(apsrc=in("waps"))^(protocol="http)]:waypoints("http-proxy"); [(apsrc=in("waps"))]:deny; []:allow;#Default-on:bydefaultallowows Figure4:AsamplepolicyleusingPol-Eth Unfortunately,intoday'smulti-useroperatingsystems,itisdif-cultfromanetworkperspectivetoattributeoutgoingtrafctoaparticularuser.InEthane,ifmultipleusersareloggedintothesamemachine(andnotidentiablefromwithinthenetwork),Ethaneap-pliestheleastrestrictiveactiontoeachoftheows.Thisisanobviousrelaxationofthesecuritypolicy.Toaddressthis,weareexploringintegrationwithtrustedend-hostoperatingsystemstoprovideuser-isolationandidentication(forexample,byprovid-ingeachuserwithavirtualmachinehavingauniqueMAC).4.3PolicyExampleFigure4containsaderivativeofthepolicywhichgovernscon-nectivityforouruniversitydeployment.Pol-Ethpolicylesconsistoftwoparts—groupdeclarationsandrules—separatedbya`%%'delimiter.Inthispolicy,allowswhichdonototherwisematcharulearepermitted(bythelastrule).Serversarenotallowedtoinitiateconnectionstotherestofthenetwork,providingprotectionsimilartoDMZstoday.Phonesandcomputerscannevercommu-nicate.Laptopsareprotectedfrominboundows(similartotheprotectionprovidedbyNAT),whileworkstationscancommunicatewitheachother.GuestusersfromwirelessaccesspointsmayonlyuseHTTPandmustgothroughawebproxy,whileauthenticatedusershavenosuchrestrictions.4.4ImplementationGivenhowfrequentlynewowsarecreated—andhowfastde-cisionsmustbemade—itisnotpracticaltointerpretthenetworkpolicy.Instead,weneedtocompileit.ButcompilingPol-Ethisnon-trivialbecauseofthepotentiallyhugenamespaceinthenet-work:Creatingalookuptableforallpossibleowsspeciedinthepolicywouldbeimpractical.OurPol-Ethimplementationcombinescompilationandjust-in-timecreationofsearchfunctions.Eachruleisassociatedwiththeprinciplestowhichitapplies.Thisisaone-timecost,performedatstartupandoneachpolicychange.Thersttimeasendercommunicateswithanewreceiver,acus-tompermissioncheckfunctioniscreateddynamicallytohandleall Figure5:Frequencyofow-setuprequestspersecondtoCon-trollerovera10-hourperiod(top)and4-dayperiod(bottom). Figure6:Flow-setuptimesasafunctionofControllerload.Packetsizeswere64B,128Band256B,evenlydistributed. agecostofanupdateona3,000nodetopologyis10ms.Inthefollowingsectionwepresentananalysisofow-setuptimesundernormaloperationandduringlinkfailure.5.3DeploymentOurEthaneprototypeisdeployedinourdepartment's100Mb/sEthernetnetwork.WeinstalledelevenwiredandeightwirelessEthaneSwitches.Therearecurrentlyapproximately300hostsonthisEthanenetwork,withanaverageof120hostsactiveina5-minutewindow.Wecreatedanetworkpolicytocloselymatch—andinmostcasesexceed—theconnectivitycontrolalreadyinplace.WepiecedtogethertheexistingpolicybylookingattheuseofVLANs,end-hostrewallcongurations,NATsandrouterACLs.Wefoundthatoftentheexistingcongurationlescontainedrulesnolongerrelevanttothecurrentstateofthenetwork,inwhichcasetheywerenotincludedintheEthanepolicy.Briey,withinourpolicy,non-servers(workstations,laptops,andphones)areprotectedfromoutboundconnectionsfromservers,whileworkstationscancommunicateuninhibited.Hoststhatcon-necttoanEthaneSwitchportmustregisteraMACaddress,butrequirenouserauthentication.WirelessnodesprotectedbyWPAandapassworddonotrequireuserauthentication,butifthehostMACaddressisnotregistered(inournetworkthismeanstheyareaguest),theycanonlyaccessasmallnumberofservices(HTTP,HTTPS,DNS,SMTP,IMAP,POP,andSSH).Ouropenwirelessaccesspointsrequireuserstoauthenticatethroughtheuniversity-widesystem.TheVoIPphonesarerestrictedfromcommunicatingwithnon-phonesandarestaticallyboundtoasingleaccesspointto Figure7:ActiveowsforLBLnetwork[19]. Figure8:Flow-requestrateforStanfordnetwork. preventmobility(forE911locationcompliance).Ourpolicyleis132lineslong.6.PERFORMANCEANDSCALABILITYDeployingEthanehastaughtusalotabouttheoperationofacentrally-managednetwork,anditenabledustoevaluatemanyas-pectsofitsperformanceandscalability,especiallywithrespecttothenumbersofusers,end-hosts,andSwitches.Westartbylook-ingathowEthaneperformsinournetwork,andthen,usingourmeasurementsanddatafromothers,wetrytoextrapolatetheper-formanceforlargernetworks.Inthissection,werstmeasuretheController'sperformanceasafunctionoftheow-requestrate,andwethentrytoestimatehowmanyow-requestswecanexpectinanetworkofagivensize.Thisallowsustoanswerourprimaryquestion:HowmanyControllersareneededforanetworkofagivensize?WethenexaminethebehaviorofanEthanenetworkunderControllerandlinkfailures.Finally,tohelpdecidethepracticalityandcostofSwitchesforlargernetworks,weconsiderthequestion:HowbigdoestheowtableneedtobeintheSwitch?6.1ControllerScalabilityRecallthatourEthaneprototypeiscurrentlyusedbyapprox-imately300hosts,withanaverageof120hostsactiveina5-minutewindow.Fromthesehosts,wesee30-40newowrequestspersecond(Figure5)withapeakof750owrequestspersec-ond.9Figure6showshowourControllerperformsunderload:forupto11,000owspersecond—greaterthanthepeaklaodweobserved—owsweresetupinlessthan1.5millisecondsintheworstcase,andtheCPUshowednegligibleload.OurresultssuggestthatasingleControllercouldcomfortablyhandle10,000newowrequestspersecond.Wefullyexpectthisnumbertoincreaseifweconcentratedonoptimizingthedesign.Withthisinmind,itisworthaskingtohowmanyend-hoststhisloadcorresponds.Weconsideredtworecentdatasets:Onefroman8,000-hostnet-workatLBL[19]andonefroma22,000-hostnetworkatStanford.Asisdescribedin[12],thenumberofmaximumoutstandingows 9Samplesweretakenevery30seconds. Figure9:Activeowsthroughtwoofourdeployedswitches Failures 0 1 2 3 4 Completiontime 26.17s 27.44s 30.45s 36.00s 43.09s Table1:CompletiontimeforHTTPGETsof275lesduringwhichtheprimaryControllerfailszeroormoretimes.Resultsareaveragedover5runs. inthetracesfromLBLneverexceeded1,200persecondacrossallnodes(Figure7).TheStanforddatasethasamaximumofunder9,000newow-requestspersecond(Figure8).Perhapssurprisingly,ourresultssuggestthatasingleControllercouldcomfortablymanageanetworkwithover20,000hosts.In-deedowsetuplatenciesforcontinuedloadofupto6,000/sarelessthan.6ms,equivalenttotheaveragelatencyofaDNSrequestwithintheStanfordnetwork.Flowsetuplatenciesforloadunder2,000requestspersecondare.4ms,thisisroughlyequivalenttotheaverageRTTbetweenhostsindifferentsubnetsonourcampusnetwork.Ofcourse,inpractice,therulesetwouldbelargerandthenum-berofphysicalentitiesgreater.Ontheotherhand,theeasewithwhichtheControllerhandlesthisnumberofowssuggeststhereisroomforimprovement.ThisisnottosuggestthatanetworkshouldrelyonasingleController;weexpectalargenetworktodeployseveralControllersforfault-tolerance,usingtheschemesoutlinedin§3.5,oneofwhichweexaminenext.6.2PerformanceDuringFailuresBecauseourControllerimplementscold-standbyfailurerecov-ery(see§3.5),aControllerfailurewillleadtointerruptionofser-viceforactiveowsandadelaywhiletheyarere-established.Tounderstandhowlongittakestoreinstalltheows,wemeasuredthecompletiontimeof275consecutiveHTTPrequests,retrieving63MBintotal.Whiletherequestswereongoing,wecrashedtheControllerandrestarteditmultipletimes.Table1showsthatthereisclearlyapenaltyforeachfailure,correspondingtoaroughly10%increaseinoverallcompletiontime.Thiscanbelargelyeliminated,ofcourse,inanetworkthatuseswarm-standbyorfully-replicatedControllerstomorequicklyrecoverfromfailure(see§3.5).LinkfailuresinEthanerequirethatalloutstandingowsre-contacttheControllerinordertore-establishthepath.Ifthelinkisheav-ilyused,theControllerwillreceiveastormofrequests,anditsperformancewilldegrade.Wecreatedatopologywithredundant Figure10:Round-triplatenciesexperiencedbypacketsthroughadiamondtopologyduringlinkfailure. paths—sothenetworkcanwithstandalink-failure—andmeasuredthelatenciesexperiencedbypackets.Failuresweresimulatedbyphysicallyunpluggingalink;ourresultsareshowninFigure10.Inallcases,thepathreconvergesinunder40ms,butapacketcouldbedelayeduptoasecondwhiletheControllerhandlestheurryofrequests.OurnetworkpolicyallowsformultipledisjointpathstobesetupbytheControllerwhentheowiscreated.Thisway,convergencecanoccurmuchfasterduringfailure,particularlyiftheSwitchesdetectafailureandfailovertousingthebackupow-entry.Wehavenotimplementedthisinourprototype,butplantodosointhefuture.6.3FlowTableSizingFinally,weexplorehowlargetheowtableneedstobeintheSwitch.Ideally,theSwitchcanholdallofthecurrentlyactiveows.Figure9showshowmanyactiveowswesawinourEthanedeployment;itneverexceeded500.Withatableof8,192entriesandatwo-functionhash-table,weneverencounteredacollision.AsdescribedearlierinFigure7,theLBLnetworkdidnotencountermorethan1,200owsintheir8,000hostnetwork.Inpractice,thenumberofongoingowsdependsonwheretheSwitchisinthenetwork.Switchesclosertotheedgewillseeanumberofowsproportionaltothenumberofhoststheyconnectto(i.e.,theirfanout).OurdeployedSwitcheshaveafanoutoffourandsawnomorethan500ows;wemightexpectaSwitchwithafanoutof,say,64toseeatmostafewthousandactiveows.(Itshouldbenotedthatthisisaveryconservativeestimate,giventhesmallnumberofowsinthewholeLBLnetwork.)ASwitchatthecenterofanetworkwilllikelyseemoreactiveows,andsoweassumeitwillseeallactiveows.FromthesenumbersweconcludethataSwitch—forauniversity-sizednetwork—shouldhaveowtablecapableofholding8K–16Kentries.Ifweassumethateachentryis64B,suchatablerequiresabout1MBofstorage,orasmuchas4MBifweuseatwo-wayhashingscheme[9].AtypicalcommercialenterpriseEth-ernetswitchtodayholds1millionEthernetaddresses(6MB,butlargerifhashingisused),1millionIPaddresses(4MBofTCAM),1-2millioncounters(8MBoffastSRAM),andseveralthousandACLs(moreTCAM).Thus,thememoryrequirementsofanEthaneSwitcharequitemodestincomparisontotoday'sEthernetswitches.TofurtherexplorethescalabilityoftheController,wetesteditsperformancewithsimulatedinputsinsoftwaretoidentifyover-heads.TheControllerwasconguredwithapolicyleof50rulesand100registeredprinciples;routeswereprecalculatedandcached.Undertheseconditions,thesystemcouldhandle650,845bindeventspersecondand16,972,600permissioncheckspersecond.The complexityofthebindeventsandpermissionchecksisdependentontherulesinuse,which,intheworstcase,growslinearlywiththenumberofrules.7.ETHANE'SSHORTCOMINGSWhentryingtodeployaradicallynewarchitectureintolegacynetworks—withoutchangingtheend-host—weencountersomestum-blingblocksandlimitations.Thesearethemainissuesthatarose:BroadcastandServiceDiscovery.Broadcastdiscoveryprotocols(ARP,OSPFneighbordiscovery,etc.)wreakhavoconenterprisenetworksbygeneratinghugeamountsofoverheadtrafc[17,20];onournetwork,theseconstitutedover90%oftheows.OneofthelargestreasonsforVLANsistocontrolthestormsofbroadcasttrafconenterprisenetworks.Hostsfrequentlybroadcastmessagestothenetworktotryandndanaddress,neighbor,orservice.Un-lessEthanecaninterprettheprotocolandrespondonitsbehalf,itneedstobroadcasttherequesttoallpotentialresponders;thisin-volvescreatinglargenumbersofowentries,anditleadstolotsoftrafcwhich—ifmalicious—hasaccesstoeveryend-host.Broad-castdiscoveryprotocolscouldbeeliminatediftherewasastandardwaytoregisteraservicewhereitcaneasilybefound.SANEpro-posedsuchascheme[12],andinthelong-term,webelievethisistherightapproach.Application-layerrouting.AlimitationofEthaneisthatithastotrustend-hostsnottorelaytrafcinviolationofthenetworkpolicy.EthanecontrolsconnectivityusingtheEthernetandIPaddressesoftheend-points,butEthane'spolicycanbecompromisedbycom-municationsatahigherlayer.Forexample,ifAisallowedtotalktoBbutnotC,andifBcantalktoC,thenBcanrelaymessagesfromAtoC.ThiscouldhappenatanylayerabovetheIPlayer,e.g.,aP2Papplicationthatcreatesanoverlayattheapplicationlayer,ormulti-homedclientsthatconnecttomultiplenetworks.Thisisahardproblemtosolve,andmostlikelyrequiresachangetotheoperatingsystemandanyvirtualmachinesrunningonthehost.Knowingwhattheuserisdoing.Ethane'spolicyassumesthatthetransportportnumbersindicatewhattheuserisdoing:port80meansHTTP,port25isSMTP,andsoon.Colludingmali-cioususersorapplicationscanfoolEthanebyagreeingtousenon-standardportnumbers.Anditiscommonfor“good”applicationstotunnelapplicationsoverports(suchasport80)thatarelikelytobeopeninrewalls.Tosomeextent,therewillalwaysbesuchproblemsforamechanismlikeEthane,whichfocusesonconnec-tivitywithoutinvolvementfromtheend-host.Intheshort-term,wecan,anddo,insertapplicationproxiesalongthepath(usingEthane'swaypointmechanism).SpoongEthernetaddresses.EthaneSwitchesrelyonthebind-ingbetweenauserandEthernetaddressestoidentifyows.IfauserspoofsaMACaddress,itmightbepossibletofoolEthaneintodeliveringpacketstoanend-host.ThisiseasilypreventedinanEthane-onlynetworkwhereeachSwitchportisconnectedtoonehost:TheSwitchcandroppacketswiththewrongMACad-dress.Iftwoormoreend-hostsconnecttothesameSwitchport,itispossibleforonetomasqueradeasanother.Asimplesolutionistophysicallypreventthis;amorepracticalsolutioninlargernetworksistouse802.1Xinconjunctionwithlink-levelencryptionmecha-nisms,suchas802.1AE,tomoresecurelyauthenticatepacketsandaddresses.8.RELATEDWORKEthaneembracesthe4D[14]philosophyofsimplifyingthedata- planeandcentralizingthecontrol-planetoenforcenetwork-widegoals[21].Ethanedivergesfrom4Dinthatitsupportsane-grainedpolicy-managementsystem.Webelievethatpolicydeci-sionscanandshouldbebasedonows.WealsobelievethatbymovingallowdecisionstotheController,wecanaddmanynewfunctionsandfeaturestothenetworkbysimplyupdatingtheCon-trollerinasinglelocation.Ourworkalsoshowsthatitispossible—webelieveforthersttime—tosecurelybindtheentitiesinthenet-worktotheiraddresses,andthentomanagethewholenamespacewithasinglepolicy.IpsilonNetworksproposedcachingIProutingdecisionsasows,inordertoprovideaswitched,multi-servicefastpathtotraditionalIProuters[18].Ethanealsousesowsasaforwardingprimitive.However,Ethaneextendsforwardingtoincludefunctionalityuse-fulforenforcingsecurity,suchasaddressswappingandenforcingoutgoinginitiatedowsonly.Indistributedrewalls[15],policyisdeclaredcentrallyinatopologyindependentmannerandenforcedateachend-host.Inadditiontotheauditingandmanagementsupport,Ethanediffersfromthisworkintwomajorways.First,inEthaneend-hostscan-notbetrustedtoenforceltering.Thismistrustisalsoextendedtothersthopswitch.Withper-switchenforcementofeachow,Ethaneprovidesmaximaldefenseindepth.Secondly,muchofthepowerofEthaneistoprovidenetworklevelguarantees,suchaspolicyimposedwaypoints.Thisisnotpossibletodothroughend-hostlevellteringalone.Pol-Eth,Ethane'spolicylanguage,isinspiredbypredicaterout-ing(PR)[22].PRuniesroutingandltering;asetofpredicatesdescribesallconnectivity.Pol-Ethextendsthismodelbymakingusersrst-classobjects,declaringpredicatesoverhigh-levelnames,andprovidingsupportforgroupdeclarationandinclusion,multipleconnectivityconstraints,andarbitraryexpressions.VLANsarewidelyusedinenterprisenetworksforsegmenta-tion,isolation,andtoenforcecoarse-grainpolicies;andtheyarecommonlyusedtoquarantineunauthenticatedhostsorhostswith-outhealth“certicates”[3,6].VLANsarenotoriouslydifculttouse,requiringmuchhand-holdingandmanualconguration;webelieveEthanecanreplaceVLANsentirely,givingmuchsimplercontroloverisolation,connectivity,anddiagnostics.ThereareanumberofIdentity-BasedNetworking(IBN)cus-tomswitchesavailable(e.g.,[4])orsecureAAAservers(e.g.,[5]).Theseallowhigh-levelpolicytobedeclared,butaregenerallypointsolutionswithlittleornocontroloverthenetworkdata-path(ex-ceptasachoke-point).Severalofthemrelyontheend-hostforenforcement,whichmakesthemvulnerabletocompromise.9.CONCLUSIONSOneofthemostinterestingconsequencesofbuildingaprototypeisthatthelessonsyoulearnarealwaysdifferent—andusuallyfarmore—thanwereexpected.WithEthane,thisismostdenitelythecase:WelearnedlessonsaboutthegoodandbadpropertiesofEthane,andfoughtanumberofresduringourdeployment.Thelargestconclusionthatwedrawisthat(oncedeployed)wefounditmucheasiertomanagetheEthanenetworkthanweex-pected.OnnumerousoccasionsweneededtoaddnewSwitches,newusers,supportnewprotocols,andpreventcertainconnectivity.Oneachoccasionwefounditnaturalandfasttoaddnewpolicyrulesinasinglelocation.Thereisgreatpeaceofmindtoknowingthatthepolicyisimplementedattheplaceofentryanddeterminestheroutethatpacketstake(ratherthanbeingdistributedasasetoflterswithoutknowingthepathsthatpacketsfollow).Byjournal-ingallregistrationsandbindings,wewereabletoidentifynumer-ousnetworkproblems,errantmachines,andmaliciousows—and