Your Server as a Function Marius Eriksen Twitter Inc - PDF document

Alloperationsreturningfuturesareexpectedtobeasynchronous,thoughthisisnotenforced.DependentcompositionItiscommontosequencetwoasyn-chronousoperationsduetoadatadependency.Forexample,asearchenginefrontend,inordertoprovidepersonalizedsearchre-sults,mightconsultauserservicetorewritequeries.Given:defrewrite(user:String,query:String):Future[String]defsearch(query:String):Future[Set[Result]]then,toperformoursearch,werstneedtoinvokerewritetoretrievethepersonalizedquery,whichisthenusedasaparametertosearch.Wecanphrasethiscombinedoperationasafuturetransfor-mation:theoperationevaluatestothefuturethatrepresentstheresultofrewrite,appliedtosearch.ThesetransformationsarelooselyanalogoustoUnixpipes[18].Withanimaginedsyntax,wemightwritedefpsearch(user,query)=rewrite(user,query)|search( ); beingtheplaceholderargumentfortheresultfromthe“pipe.”TheflatMapcombinatorperformsthiskindoftransformation:traitFuture[T]{defflatMap[U](f:T�=Future[U]):Future[U]...}(ThetypeT�=Future[U]denotesaunaryfunctionwhosear-gumentisoftypeTandwhoseresultisoftypeFuture[U].)ThusflatMap'sargumentisafunctionwhich,whenthefuturesucceeds,isinvokedtoproducethedependentfuture.flatMap,aswithallfuturecombinators,isasynchronous:itreturnsimmediatelywithafuturerepresentingtheresultofthecompositeoperation.PersonalizedsearchisastraightforwardapplicationofflatMap:defpsearch(user:String,query:String)=rewrite(user,query)flatMap{pquery�=search(pquery)}(Scalaprovides“inx”notationforunarymethods;parenthesesaroundthemethodargumentareomitted,soisthedotformethoddereference.Functionliteralsareintroducedwithfparam�=exprg.)psearchreturnsaFuture[Set[Result]],thusvalresult:Future[Set[Result]]=psearch("marius","finagle")issuesthepersonalizedsearchfor“nagle;”resultisthefuturerepresentingtheresultofthecomposedoperation.Inthisexample,flatMapisusedtoresolveadatadependencybetweensearchandrewrite;psearchmerelyexpressesthis,withoutspecifyinganexecutionstrategy.HandlingerrorsflatMapshort-circuitscomputationwhentheouterfuturefails:thereturnedfutureisfailedwithoutinvokingthegivenfunctiontoproduceadependentfuture.Indeed,itstypebearswitnesstoitssemantics:whenaFuturefails,itdoesnothaveavaluetopresenttothefunctionproducingthedependentfuture.Theseerrorsemanticsareanalogoustotraditional,stack-basedexceptionsemantics.Itisoftenusefultorecoverfromsucherrors,forexampleinordertoretryanoperation,ortoprovideafallbackvalue.Therescuecombinatorprovidesthis;whereasflatMapoperatesoversuccessfulresults,rescueoperatesoverfailures.traitFuture[T]{...defrescue[B](f:PartialFunction[Throwable,Future[B]]):Future[B]}WhileflatMapdemandsatotalfunction,rescueacceptspartialfunctions,allowingtheprogrammertohandleonlyasubsetofpos-sibleerrors.Forexample,wecanmodifypsearchfromabovetoskippersonalizationifthequeryrewritingsystemfailstocompletewithinadeadline,sothatadegradedresultisreturnedinsteadofafailure:defpsearch(user:String,query:String)=rewrite(user,query).within(50.milliseconds)rescue{case_:TimeoutError�=Future.value(query)}flatMap{pquery�=search(pquery)}(fcase...isapartialfunctionliteralinScala;itmaycon-tainmultiplecaseclauses.withinisamethodonFuturethatreturnsanewFuturewhicheithercompleteswithinthegivendu-ration,orfailswithatimeouterror.)Ifthefuturereturnedfromrewrite(..).within(..)fails,andthepartialfunctionisde-nedforthespecicfailure—inthiscase,wematchontheexcep-tiontypeTimeoutError—theerrorisrecoveredbysupplyingtheoriginalqueryvalue.Sinceeachcombinatorreturnsafuturerep-resentingtheresultofthecompositeoperation,wecanchainthemtogetherasinthisexample.Thisstyleisidiomatic.ComposingmultipledependenciesServerscommonlyperform“scatter-gather,”or“fan-out”operations.Theseinvolveissuingre-queststomultipledownstreamserversandthencombiningtheirresults.Forexample,Twitter'ssearchengine,Earlybird[5],splitsdataacrossmultiplesegments;afrontendserveranswersqueriesbyissuingrequeststoareplicaofeachsegment,thencombinestheresults.Thecollectcombinatorresolvesmultipledependencies.ForsomevaluetypeA,collectconvertsasequenceoffuturesintoafutureofasequenceofA-typedvalues:defcollect[A](fs:Seq[Future[A]]):Future[Seq[A]](SeqisScala'sgenericcontainerforsequences.)Asketchofascatter-gatheroperationfollows.Givenamethodtoqueryseg-ments,defquerySegment(id:Int,query:String):Future[Set[Result]]weusecollecttorelatemultipledependencies;thesearchmethodfromthepreviousexamplecanthusbedened:defsearch(query:String):Future[Set[Result]]={valqueries:Seq[Future[Result]]=for(id0untilNumSegments)yield{querySegment(id,query)}collect(queries)flatMap{results:Seq[Set[Result]]&#x--52;倀=Future.value(results.flatten.toSet)}}AswithflatMap,errorsinanyfuturepassedasanargumenttocollectpropagateimmediatelytothecomposedfuture:shouldanyofthefuturesreturnedbyquerySegmentfail,thecollectedfuturewillfailimmediately.YourServerasaFunction(Preprint)22013/9/27 thependingfuture(asreturnedbytheserver-suppliedService).ThisallowsustointerruptarequestinourfrontendHTTPserver,cancelingallongoingworkonbehalfofthatrequestthroughoutourdistributedsystems.Aswithourtracingsystem,thiswasim-plementedwithoutanyAPImodicationsorotherchangestousercode.Whileinterruptsviolatethepuredataowmodelpresentedbyfutures,consumersarestilloblivioustotheirproducers.Interruptsareadvisory,anddonotdirectlyaffectthestateofthefuture.Interruptsarenotwithoutproblems.Theyintroducenewseman-ticcomplications:Shouldcombinatorspropagateinterruptstoallfutures?Oronlytheoutstandingones?Whatifafutureissharedbetweenmultipleconsumers?Wedon'thavegreatanswerstothesequestions,butinpracticeinterruptsareusedrarely,andthenalmostexclusivelybyFinagle;wehavenotencounteredanyproblemswiththeirsemanticsortheirimplementation.4.3FiltersFiltershaveheldtheirpromiseofprovidingclean,orthogonal,andapplication-independentfunctionality.Theyareuseduniver-sally:Finagleitselfusesltersheavily;ourfrontendwebservers—reverseHTTPproxiesthroughwhichallofourexternaltrafcows—usealargestackoflterstoimplementdifferentaspectsofitsresponsibilities.Thisisanexcerptfromitscurrentcongura-tion:recordHandletimeandThentraceRequestandThencollectJvmStatsandThenparseRequestandThenlogRequestandThenrecordClientStatsandThensanitizeandThenrespondToHealthCheckandThenapplyTrafficControlandThenvirtualHostServerFiltersareusedforeverythingfromlogging,torequestsanitiza-tion,trafccontrol,andsoon.Filtershelpenhancemodularityandreusability,andtheyhavealsoprovedvaluablefortesting.Itisquitesimpletounittesteachoftheseltersinisolation—theirinterfacesaresimpleanduniform—withoutanysetup,andwithminimalmocking.Furthermore,theyencourageprogrammerstoseparatefunctionalityintoindependentmoduleswithcleanbound-aries,whichgenerallyleadstobetterdesignandreuse.Wehavealsousedltersextensivelytoaddresslower-levelconcerns.Forexample,wewereabletoimplementasophisticatedbackuprequestmechanismusingasimplelterinabout40linesofcode(seeAppendixA).EngineersatTumblr,whoalsouseFinagle,report[16]theuseofalowlevelltertodeduplicaterequeststreams.4.4ThecostofabstractionHighlevelprogramminglanguagesandconstructsdonotcomeforfree.Futurecombinatorsallocatenewfuturesonthegarbagecollectedheap;closures,too,needtobeallocatedontheheap,sincetheirinvocationisdeferred.Whilewe'vefocusedonreducingtheallocationfootprints—andindeedcreatedmanytoolsforallocationanalysis—itisanongoingconcern.Thetaillatenciesofmostofourserversaregovernedbyminorheapgarbagecollections.Inisolation,thisimpliesonlyasmallser-vicedegradation.Howeverourlargefan-outsystemampliessucheffectsasoverallrequestlatencyisgovernedbytheslowestcom-ponent;withlargerequestdistribution—often100sofsystems—encounteringminorgarbagecollectionintherequestpathiscom-mon.DeanandBarroso[7]describesimilarexperiencesatGoogle.Afrequentsourceofunintentionalgarbagecollectionpressureistheeasewithwhichspaceleakscanbeintroducedbythein-advertentcapturingofreferencesinclosures.Thisisampliedbylong-livedoperations,forexample,closuresthataretiedtolifetimeofaconnection,andnotofarequest.Milleret.al.'sSpores[14]proposestomitigatethesetypesofleaksbygivingtheprogrammerne-grainedcontrolovertheenvironmentcapturedbyaclosure.Inmostofourservers,majorcollectionsarerare.Thisgivesrisetoanotherkindofspaceleak:ifaPromiseispromotedtothemajorheap(forexamplebecausetheoperationitrepresentstookanunexpectedlylongtime),itsreferentvalue,evenifitsusefullifetimeisminiscule,survivesuntilthenextmajorgarbagecollection.Developmentdisciplineisanimportantmitigatingfactor.Inordertoensurethatallocationregressionsaren'tintroduced,wehavedevelopedatool,JVMGCPROF[9]whichrunsregularlyalongwithourtests,providingreportsonper-requestallocationratesandlifetimes.Thisisanareaofongoingeffortwithmanyintriguingpossibil-ities.SinceFinaglecontrolslogical-to-physicalthreadmultiplex-ingandisawareofrequestboundaries,itcanbiasallocation.Thisopensupthepossibilitythat,withthecooperationoftheunderlyingJVM,wemaymakeuseofregionallocationtechniques[13].4.5Futures,Services,andFiltersatTwitterThesetechniquesareusedtogetherwithourRPCsystem[10],throughoutourproductionruntimesystems.Almostallofourmod-ernserversoftwareisimplementedinthisway,includingourfron-tendservingsystem,applicationservers,webcrawlingsystems,databasesystems,andfan-outanddatamaintenancesystems.Thesystemsareemployedinamajorityofmachinesacrossmultipledatacenters.We'vefoundthesetechniquesexcelespeciallyinmiddlewareservers,whoseinterfaceisaServiceandwhosedependenciesareotherServices.Suchmiddlewarereducestoeffectivelybig“futuretransformers”:it'scommontoexpresstheentiresysteminadeclarativefashion.Theadvantagesofuniformityextendbeyondindividualserversoftware.Statistics,tracingdata,andotherruntimeinformationisobservedwiththeuseofacommonsetoflters.Theseinturnexportsuchruntimeinformationinauniformly,sothatoperatorscanmonitoranddiagnoseoursystemswithoutknowingspecics.OurofinedataprocessingsystemshavenotyetadoptedthesetechniquesastheyareoftenbuiltontopofotherframeworkssuchasHadoop.5.RelatedworkLwt[22]isacooperativethreadinglibraryforOCamlwhosechiefabstraction,thelightweightthread,issimilartoourFuture.Dataowprogramminglanguages[8,21,23]alsoemphasizedependenciesbetweencomputations,performingconcurrentgraphreductionintheirruntimes.However,dataowconcurrencyde-mandsdeterminacy,requiring,amongotherthings,timinginde-pendenceandfreedomfromnondeterminateerrors(mostsuchlan-guagesrequirefreedomfromanyerrors).Thusinitsrawform,dataowlanguagesareunsuitableforsystemsprogramming.Royet.al.[19]proposeintroducingnondeterminateportstosplitadataowprogramintopure(determinate)parts,connectedbynon-deterministicchannels.Thisisintriguing,butinsystemsprogram-ming,nearlyallconcurrencyisnondeterministic.(Indeed,youcouldarguethatdeterministicconcurrencyisbetterdescribedasparallelism.)Haskell[2]andGo[1]providecheapuser-spacethreads,reduc-ingtheoverheadofthread-basedconcurrency.Theseruntimesman-agethreadsasacheapresource,andfreestheprogrammerfromtheYourServerasaFunction(Preprint)52013/9/27 A.BackuprequestlterclassBackupRequestFilter[Req,Rep](quantile:Int,range:Duration,timer:Timer,statsReceiver:StatsReceiver,history:Duration,stopwatch:Stopwatch=Stopwatch)extendsSimpleFilter[Req,Rep]{require(quantile�0&&quantile100)require(range1.hour)private[this]valhisto=newLatencyHistogram(range,history)private[this]defcutoff()=histo.quantile(quantile)private[this]valtimeouts=statsReceiver.counter("timeouts")private[this]valwon=statsReceiver.counter("won")private[this]vallost=statsReceiver.counter("lost")private[this]valcutoffGauge=statsReceiver.addGauge("cutoff_ms"){cutoff().inMilliseconds.toFloat}defapply(req:Req,service:Service[Req,Rep]):Future[Rep]={valelapsed=stopwatch.start()valhowlong=cutoff()valbackup=if(howlong==Duration.Zero)Future.neverelse{timer.doLater(howlong){timeouts.incr()service(req)}flatten}valorig=service(req)Future.select(Seq(orig,backup))flatMap{case(Return(res),Seq(other))&#x-525;=if(othereqorig)lost.incr()else{won.incr()histo.add(elapsed())}other.raise(BackupRequestLost)Future.value(res)case(Throw(_),Seq(other))&#x-525;=other}}}Thebackuplterissuesasecondaryrequestifaresponsehasnotarrivedfromtheprimaryrequestwithinagiventime,parameterizedbytheobservedrequestlatencyhistogram.Whenaresponsearrives—thersttoarrivesatisfythefuturereturnedbyFuture.select—theotherrequestiscancelledinordertoavoidunnecessarywork.(ServicesfurnishedbyFinagleareloadbalanced,sothatrepeatedinvocationstoservicearelikelytobedispatchedtodifferentphysicalhosts.)YourServerasaFunction(Preprint)72013/9/27