lastnamekitedu Abstract Due to the advent of eventually consistent storage systems consistency has become a focus of research Still a clear overview of con sistency in distributed systems is missing In this work we de64257ne and describe consistency ID: 70788
Download Pdf The PPT/PDF document "Consistency in Distributed Storage Syste..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
2DavidBermbachandJornKuhlenkampcombinethiswithourpreviousworkonconsistencybenchmarkinganddescribecontinuousmetricsandsetupsforexperimentalconsistencymeasurements.Themaincontributionofthisworkis,hence:{Acomprehensiveoverviewofconsistencymodelsandguarantees,includingdiscussionswhyparticularguaranteesareusefulindi erentusecases.{Ananalysishowconsistencymodelscanbeorderedbytheirstrictnessandhowclient-centricguaranteesrelatetodata-centricmodels.{Adiscussionofdi erentmetricstodescribeconsistencyguaranteesaswellasabriefanalysisofexistingbenchmarkingapproaches.2DenitionsThetermconsistencyisderivedfromthelatinwordconsisterewhichmeans\standingtogether"oralso\stoppingtogether".Hence,consistencygenerallydescribesrelationshipsbetweenitemsthataresomehowconnected.Whencon-sideringconsistencyofdata,aconsistentstaterequiresthatallrelationshipsbetweendataitemsandreplicasareastheyshouldbe,i.e.,thatthedatarep-resentationiscorrect.Thisfocusonthecorrectnesscanbeseeninboththedatabaseaswellasthedistributedsystemscommunity{butondi erentlevels.2.1DatabaseSystemsResearchersfromthe(relational)databasecommunityfocusonconsistencyinthecontextofACIDguarantees.Asetofoperationsisabstractedintoatransac-tionthatwillexecuteentirelyornotatall(atomicity).Ifitexecutes,itschangeswillbepermanentlyvisible(durability).Multipletransactionsmayexecutecon-currentlybutusesuitablemechanismstoshowthesamebehaviorasifalltrans-actionswereexecutedaccordingtosomeglobalserializableorder.Thiscanbedoneinvariousdegreesofisolation1rangingfromtrueserializability(whichisrarelyo eredinactualproducts[5])toreaduncommitted.Transactionsguaran-teethatthedatabasealwaysadherestoaglobalschema(independentofwhetherthedatabaseisreplicatedornot)whereallintegrityconstraints,asetofcondi-tionsandrequirements,areobserved[15](consistency).Hence,theconsistencyfocusofdatabasesystemsisontherelationshipsbe-tweendataitemsandtheoverallcorrectnessoftheentiredatabase.Theycanbeguaranteedinadistributedsettingbutitisexpensivetodosoasconsistencyandisolationaretypicallyguaranteedvialockingmechanismswhichcreateanextensivecommunicationoverheadinadistributedsetting.Thisisalsocausedbythefact,thattransactionswereintendedas\asimpleprogrammingabstrac-tiontocopewithconcurrentexecutions,ratherthantoaddressthechallengesofadistributedsetting."[33]. 1Isolationdescribesthedegreetowhichconcurrenttransactionsareawareofeachother,e.g.,byaccessingthesamedataitems. 4DavidBermbachandJornKuhlenkamp3.2ConsistencyModelsandImplementationsBothdata-centricandclient-centricconsistencyguaranteeshavetwodimensions:orderingandstaleness3.Stalenessdescribeshowmuchagivenreplicaislaggingbehind,eitherexpressedintermsoftime(t-visibility)orversions(k-staleness)[4].Again,k-stalenessisafunctionoft-visibilityandtheupdatepatternsoftheap-plicationsothat,forapplication-independentinformation,t-visibilitysucestocharacterizethestalenessguaranteesofastoragesystem.Low,boundedstale-nessvaluescanoftenbetoleratedbyapplicationsaslongasthecorrespondingreal-worldeventswouldhavethesameorhigherstalenessvalueswithoutanITsystem.Forexample,whenpersonAwiresmoneytopersonB,theaccountofAwillbechargedrightaway.PersonBincontrastmightnotbecreditedforsometime.IntheEUthistimewindowislimitedtothreedayswhichisfarlongerthananyreplicasynchronizationprotocolmighttake.Hence,smallstalenessvalueswilloftennotbenoticed.Orderingontheotherhandismorecritical.Inasettingwithstrictconsis-tency,allrequestsmustbeexecutedonallreplicasintheirchronologicalorderwhichishardtoimplementindistributeddatabasesduetoclocksynchroniza-tionissuesasthereplicaserversmightdisagreeontheactualchronologicalorderofevents.Thestandarddatabasemechanismoflockingo erspoorperformancelevelsinadistributedsetting.Basedonthis,data-centricconsistencymodelsexistthatrelaxcertainorderingrequirementswhilekeepingthosethatarees-sentialtoapplications.Thesemodelscanbeorderedbythe\strictness"oftheirguarantees.Client-centricconsistencymodelstakeadi erentapproach:Whiletherewillalmostcertainlybecrosse ectsbetweenthemodels,theguaranteesitselfaredisjunctintheirpromisesandcomplementeachother.Wewillstartbydescribingclient-centricconsistencymodelsbeforecontinuingtodata-centricmodelsandhowthosetwoarerelated.Client-centricConsistencyTherstmodel,MonotonicReadConsistency(MRC),guaranteesthataclientthathasreadaversionnwillthereafteralwaysreadversionsn[34,37].Thisishelpfulasfromanapplicationperspectivedatavisibilitymightnotbeinstantaneousbutversionsatleastbecomevisibleinchronologicalorder,i.e.,thesystemnever\goesbackward"intime.Forexample,imaginepersonBfromourbankscenarioabove.IfthispersonrstseesthecreditedamountonhisbankaccountstatementandthentriestotransferthemoneytoapersonCwhichfailsdueto\insucientfunds",thiswillatleastcauseseverecustomerirritationifnotmore.ReadYourWritesConsistency(RYWC)guaranteesthataclientthathaswrittenaversionnwillthereafteralwaysbeabletoreadaversionthatisatleastasnewasn,i.e.,n[34,37].Thishelps,forexample,toavoiduserir-ritationwhenpersonAcheckshisbankaccountstatement,doesnotseethe 3YuandVahdat[39]proposeanadditionaldimensionnumericalerrortodescribereplicadi erencesbasedonthesemanticsoftherespectivedataitem.Fromourpointofview,thisisrstnotalwaysapplicableandsecondanumericalerrorisessentiallyafunctionofordering,stalenessandapplicationaccesspatterns. 6DavidBermbachandJornKuhlenkampIntermsofclient-centricconsistencyguarantees,ECoftenfulllstheseguar-anteesforamajorityofrequestsbutdoesnotguaranteetodoso.Asanexample,AmazonS34currentlydeliversMRCforabout95%ofallrequestswhereasitstillviolatedMRCinabout12%ofallrequestsin2011[8].WhiletherearecertainlysomeusecaseswhereECcannotbeapplied,itoftensucesastherealworlditselfisinherentlyeventuallyconsistent.Thedi erenceis,thatmorecon ictresolutionisnecessaryattheapplicationlayer[16]requiringahigherskillsetfromapplicationdevelopers.Insteadofpessimisticallylockingdataitems\guessesandapologies"areused[22].CausalConsistency(CC)isthestrictestlevelofconsistencythatcanbeachievedinanalwaysavailablestoragesystem[30]basedonthetradeo softheCAPtheorem[18].Inacausallyconsistentstoragesystem,allrequeststhathaveacausalrelationshiptoanotherrequestmustbeserialized(i.e.,executed)inthesameorderonallreplicaswhileunrelatedrequestsmaybeserializedinarbitraryorder.Arequestr2causallydependsonarequestr1{ifbothrequestsareissuedbythesameclientandr1wasreceivedatthestoragesystembeforer2,{ifr2isareadthatreturnstheresultofr1whichisanupdateor{ifthereisatransitiverelationbetweenbothrequests[34,37,9].Ofcourse,CCcapturespotentialcausalitysothatsystemslikeCOPS[29]havetoevaluatelargedependencytreesbeforeapplyinganupdate.Thisbothaddsanoverheadandincreasesstalenessasupdatescannotbecomevisiblerightaway.Bailisetal.[3]proposetominimizethisimpactbyhavingtheapplicationexplic-itlydenedependenciesthatneedtobeconsidered.Atypicalimplementationusesvectorclockstoidentify(potential)causaldependencies.CCcanalsobedenedviatheclient-centricguaranteesdiscussedabove:Ifallfourarefullled,thesystemiscausallyconsistent[9].Itisalsopossibletocreatetheclient-sideillusionofCCwiththecombinationofversioncachingandvectorclocks[7].AsGuerraouiandHaripointout,CCdoesnotrequirereplicaconvergence[21].Convergenceisonlyassertedwhenthelatestupdateiscausallydependentonallpreviouswritessincethelastidempotentreplace-update5andstalenessisbounded.SequentialConsistency(SC)isaverystrictconsistencymodelandcannotbeachievedinalwaysavailablesystems6.Itrequiresthatallrequestsareserializedinthesameorderonallreplicasandthatrequestsbythesameclientareexecutedintheorderthattheyarereceivedbythestoragesystem[34].Whilethismodel 4aws.amazon.com/s35i.e.,somerequestlikex:=5whichdoesnotdependonanypreviousvalue.6InCConlyrequestswithcausaldependenciesmustbeexecutedinthesameorderonallreplicas.ForSC,thisextendstoallrequestssothatreplicasneedtoagreeontheorderingofrequestsfornon-causallyrelatedrequests.Thisisnotpossibleinthepresenceoffailuressothatthesystemeitherbecomesunavailableorviolatesitsconsistencymodel. 8DavidBermbachandJornKuhlenkampClient"describesthattheguaranteesfromsection3.2arefullled,whereasweuse\Global"todescribewhensuchaguaranteeisextendedtoallclientsatthesametime.Data-centricModel MRC RYWC MWC WFRC WeakConsistency N/A N/A N/A N/AEventualConsistency Often Often Often OftenCausalConsistency SingleClient SingleClient SingleClient SingleClientSequentialConsistency SingleClient SingleClient SingleClient GlobalLinearizability Global Global Global GlobalTable1.RelationshipBetweenData-centricandClient-centricConsistencyModelsOrderedbytheStrictnessoftheirGuaranteesOtherConsistencyModelsBeyondthemodelsalreadydiscussed,therearealsoafewotherconsistencymodelsthatdonotquitetthecategorizationusedsofar.Multi-dimensionalConsistency:YuandVahdat[39]introducetheconceptofaconit,aconsistencyunit,whichisathreedimensionalvectorthatdescribestolerabledeviationsfromLINalongthedimensionsstaleness,ordererrorandnu-mericalerror.Asalreadymentioned,numericalerrorisoftennotapplicableandsemanticallyoverlapswithstalenessandordererror.Whenignoringnumericalerror,theirworkbecomescomparabletotheworkofTorres-Rojasetal.(e.g.,[36,35])whocoinedthetermtimedconsistency.Timedconsistencymodelsarealsosometimesknownasdeltaconsistencyandessentiallydescribeacombinationoforderingandstalenessinthattheinconsistencywindow(denedbythetimeperiodbetweenthecommitofanupdateandreachingaconsistentstate)isbound.Thismeansthattheguaranteesofaparticularconsistencymodelarenotreachedrightawaybutratherafteraxedperiodoftimet.Ifreplicasfailtosynchronizeduringthatperiodoftime,theitembecomesunavailableuntilconsistencyhasbeenreached.ThisisparticularusefulforguaranteeingSer-viceLevelAgreements(SLAs)andincreasesthetransparencyoftheconsistencyavailabilitytrade-o .Sadly,toourknowledgenoimplementationsoftimedconsistencymodelsexistapartfromTACT[39]andtheworkofKrishnamurthyetal.[25]whoguaranteeboundsonk-staleness(basedonversioncount).Itispossible,though,tospecifyatimedversionforeachofthedata-centricconsistencymodelswheretheguaranteesbecomevisiblebeforethespeciedtimewindowisover.Inthatcase,themodelsdiscussedabovebecomeaspecialcaseoftheirtimedequivalent(i.e.,withatimewindowofinnity)whichalsoa ectsthetimelinessofclient-centricguarantees. 10DavidBermbachandJornKuhlenkamp4MeasuringConsistencyGuarantees4.1ContinuousConsistencyMetricsAccordingtothefreedictionary.com,ametricis\Astandardofmeasurement".Whenmeasuringacertainaspect,ameasurementalwayscomprisesavalueandacorrespondingunit(e.g.,fortheheightofabuildingthiscouldbethevalue\5"andtheunit\meter").Ifitisnotpossibletondtwovalueswhichdonothaveanyvalueinbetweenthem,themetriciscontinuous.Otherwisethemetricisdiscrete.Anexampleforacontinuousmetricwouldbetheheightofaperson,whereasclothingsizesareanexampleforadiscretemetric.Whentheultimategoalistocompareconsistencyguaranteesoftwostoragesystems,itisdesirabletoeitheruseacontinuousmetricoratleastuseadiscretemetricwithalargenumberofpotentialmeasurementvalues.Otherwise,itmightnotbepossibletoranksystemsaccordingtotheirconsistencyguarantees.Inthefollowing,wewilldiscussmetricsfordata-centricandclient-centricconsistency.Dependingontheperspective(storagesystemproviderorapplicationdeveloper),di erentmetricsmaybethebestt.Data-centricConsistencyMetricsZellagandKemme[40]extendtheirpre-viousworkontransactionaldatastorestonon-transactionaldatastores.Theyproposetobuildaglobaldependencygraphbasedonoperationlogsandcountcyclesinthegraphasametricfor\consistencyanomalies".ThisisadiscretemetricandoneoftheirmainassumptionsisthatthestoragesystemguaranteesatleastCCwhichisveryrestrictiveanddoesnotallowtoanalyzeconsistencyguaranteesofmostNoSQLsystemswhichonlyo erEC.Table2liststheirap-proachas\Anomalies".Rahmanetal.[32],Golabetal.[19]andAndersonetal.[2]atHewlettPackardLabsalsoproposetobuilddependencygraphsbasedonoperationlogsandtocountcyclesinthegraphasametricforconsistencyguarantees.Theydistinguishthethreepropertiessafeness,regularityandatomicityforwhichtheyeachcountviolations.AstoragesystemthathasnocyclesinitsatomicitygraphfulllsLIN.Theothertwopropertiesalsoconsiderstalenessaswellasordering.Regularityis,thus,stricterthanSCwhereasSafetycannotbecomparedtoexistingconsistencymodels.Regularitymandatesthat\areadnotconcurrentwithanywritesreturnsthevalueofthemostrecentwrite,andareadconcurrentwithsomewritesreturnseitherthevalueofthemostrecentwrite,orthevalueofoneoftheconcurrentwrites"[2].Safenessincontrastrelaxesthelastrequirementsothatreadsconcurrentwithwritesmayreturnarbitraryvalues.WedonotbelievethatthelatterguaranteeisveryhelpfulasitbasicallyrequiresLINfornon-concurrentrequestsandWeakConsistencyforconcurrentrequests.Chockleretal.[14]seemtosharethatopinion.Furthermore,real-worldsystemsmayormaynotreturnthevalueofthemostrecentwritebut,toourknowledge,nosystemexiststhatmayreturnvaluesthathaveneverbeenwritten.Allthreemetricscanalsobeexpressedask-propertyor-property(e.g.,k-atomicityand-atomicity)whichdescribesthemaximumnumberoftimeunitsorversionsa 12DavidBermbachandJornKuhlenkampStalenessisbestexpressedeitherintermsoftime(t-Visibility)orversionlag(k-Staleness)inbothcasesthecorrespondingdata-centricvalueisanupperboundfortheclient-centricone,asasystemmayemployadditionalmecha-nismstohidestalenessfromtheapplication.Forexample,inaquorumsystemwithan(N,R,W)congurationof(5,2,2)data-centrict-Visibilitywillbedeter-minedbythetimedi erencebetweenthestartoftheupdateinreplica1(orthecommittimestamp{thisdependsonwhenupdatesbecomevisible:rightawayoruponcommit)andtheendoftheupdateinreplica5.Theclient-centrict-Visibility,incontrast,isdeterminedagainbythesamestarttimestampbutendswhenreplica4completesthewriteasafterwardsnorequestwilleveragainreturntheoldvalue.Hence,data-centricstalenessvaluesareanupperboundforclient-centricstalenessvalues.Stalenesscaneitherbeexpressedasadensityfunction(probabilitydistributionofinconsistencywindowsizes)orasacumu-lativedensityfunction(probabilityofreadingfreshdatattimeunitsafterthelastupdate).Orderingisbestexpressedintermsoftheclient-centricconsistencymodels,i.e.,thelikelihoodofarequestviolatingaparticularguarantee.Table3givesanoverviewofclient-centricconsistencymetrics.Metric Staleness Ordering Continuous Discrete Unit&Description MRCViolations - X X - Prob.distr.ofviolationMWCViolations - X X - Prob.distr.ofviolationRYWCViolations - X X - Prob.distr.ofviolationWFRCViolations - X X - Prob.distr.ofviolationt-Visibility X - X - Prob.distr.oftimelagk-Staleness X - X - Prob.distr.ofversionlagTable3.OverviewofClient-centricConsistencyMetrics4.2ConsistencyBenchmarkingApproachesAfteridentifyingthemetricsmostusefulformeasuringconsistencyinthelastsection,wewillnowdescribebenchmarkingapproachesforthesemetrics.Data-centricConsistencyAlldata-centricmetricsrequireaccesstotheactualreplicasofthestoragesystem.Atestapplicationcreatesloadonthesystem.Resultsarethenachievedbyminingreplicalogswhichshouldforeachrequestcontainthefollowinginformation:startandendtimestampateachreplica,someuniquerequestidandtherequesttype(read,write).Basedonthis,itisthenpossibletocalculatet-Visibility,k-Stalenessaswellasthenumberoforderingviolationsandthecorrespondingconsistencymodel8. 8Someadditionalinformationlikethe(N,R,W)congurationforaquorumsystemmaybenecessary. 14DavidBermbachandJornKuhlenkampindistributedsystems.Next,wecontinuedwithadetaileddiscussionofdata-centricandclient-centricconsistencymodels,theirusecasesandtherelationshipsbetweenthosemodelsbeforedescribingmetricsandbenchmarkingapproachesthathelptodetermineconsistencyguaranteesofdistributedstoragesystems.References1.Abadi,D.:Consistencytradeo sinmoderndistributeddatabasesystemdesign:Capisonlypartofthestory.Computer45(2),37{42(feb2012)2.Anderson,E.,Li,X.,Shah,M.,Tucek,J.,Wylie,J.:Whatconsistencydoesyourkey-valuestoreactuallyprovide.In:HotDep(2010)3.Bailis,P.,Fekete,A.,Ghodsi,A.,Hellerstein,J.,Stoica,I.:Thepotentialdangersofcausalconsistencyandanexplicitsolution.In:ProceedingsoftheThirdACMSymposiumonCloudComputing.p.22.ACM(2012)4.Bailis,P.,Venkataraman,S.,Hellerstein,J.,Stoica,I.:Probabilisticallyboundedstalenessforpracticalpartialquorums.VLDBEndowment(2012)5.Bailis,P.:Whenis\acid"acid?rarely.http://www.bailis.org/blog/when-is-acid-acid-rarely(accessedJan28,2013)(2013)6.Baker,J.,Bond,C.,Corbett,J.,Furman,J.,Khorlin,A.,Larson,J.,Leon,J.,Li,Y.,Lloyd,A.,Yushprakh,V.:Megastore:providingscalable,highlyavailablestorageforinteractiveservices.In:ProceedingsofConferenceonInnovativeDataSystemsResearch7.Bermbach,D.,Kuhlenkamp,J.,Derre,B.,Klems,M.,Tai,S.:Amiddlewareguar-anteeingclient-centricconsistencyontopofeventuallyconsistentdatastores.In:IC2E.IEEE(2013)8.Bermbach,D.,Tai,S.:Eventualconsistency:Howsooniseventual?anevalua-tionofamazons3'sconsistencybehavior.In:Proceedingsofthe6thWorkshoponMiddlewareforServiceOrientedComputing.p.1.ACM(2011)9.Brzezinski,J.,Sobaniec,C.,Wawrzyniak,D.:Fromsessioncausalitytocausalconsistency.In:PDP(2004)10.Brzezinski,J.,Sobaniec,C.,Wawrzyniak,D.:Sessionguaranteestoachievepramconsistencyofreplicatedsharedobjects.PPAM(2004)11.Chang,F.,Dean,J.,Ghemawat,S.,Hsieh,W.,Wallach,D.,Burrows,M.,Chandra,T.,Fikes,A.,Gruber,R.:Bigtable:Adistributedstoragesystemforstructureddata.ACMTransactionsonComputerSystems(TOCS)26(2),1{26(2008)12.Chihoub,H.,Ibrahim,S.,Antoniu,G.,Perez,M.,etal.:Consistencyinthecloud:Whenmoneydoesmatter!(2012)13.Chihoub,H.,Ibrahim,S.,Antoniu,G.,Perez,M.,etal.:Harmony:Towardsauto-matedself-adaptiveconsistencyincloudstorage.In:IEEECLUSTER(2012)14.Chockler,G.,Guerraoui,R.,Keidar,I.,Vukolic,M.:Reliabledistributedstorage.Computer42(4),60{67(2009)15.Codd,E.F.:Therelationalmodelfordatabasemanagement:Version2.Addison-Wesley,Reading,Mass.(1990)16.DeCandia,G.,Hastorun,D.,Jampani,M.,Kakulapati,G.,Lakshman,A.,Pilchin,A.,Sivasubramanian,S.,Vosshall,P.,Vogels,W.:Dynamo:amazon'shighlyavail-ablekey-valuestore.In:Proc.SOSP(2007)17.Ghemawat,S.,Gobio ,H.,Leung,S.:TheGooglelesystem.ACMSIGOPSOperatingSystemsReview37(5),29{43(2003)