COS 418 Distributed Systems Lecture 14 Wyatt Lloyd Consistency Hierarchy Linearizability Sequential Consistency Causal Consistency Eventual Consistency eg RAFT eg Bayou eg Dynamo ID: 1002137
Download Presentation The PPT/PDF document "Scalable Causal Consistency" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
1. Scalable Causal ConsistencyCOS 418: Distributed SystemsLecture 14Wyatt Lloyd
2. Consistency HierarchyLinearizabilitySequential ConsistencyCausal+ ConsistencyEventual Consistencye.g., RAFTe.g., Bayoue.g., DynamoCAPPRAM 1988(Princeton)
3. Consistency HierarchyLinearizabilitySequential ConsistencyCausal+ ConsistencyEventual Consistencye.g., RAFTe.g., Bayoue.g., DynamoCAPPRAM 1988(Princeton)Part 1: More on consistencyP2: Scalable Causal Consistency
4. Last Time’s Causal+ ConsistencyPartially orders all operations, does not totally order themDoes not look like a single machineGuaranteesFor each process, ∃ an order of all writes + that process’s readsOrder respects the happens-before () ordering of operations+ replicas converge to the same stateSkip details, makes it stronger than eventual consistency
5. Causal ConsistencyWrites that are potentially causally related must be seen by all processes in same order. Concurrent writes may be seen in a different order on different processes.Concurrent: Ops not causally related
6. Causal ConsistencyP1abdP2P3Physical time ↓efgcWrites that are potentially causally related must be seen by all processes in same order. Concurrent writes may be seen in a different order on different processes.Concurrent: Ops not causally related
7. Causal ConsistencyP1abdP2P3efgcOperationsa, bb, fc, fe, fe, ga, ca, eConcurrent?NYYYNYNPhysical time ↓
8. Causal ConsistencyP1abdP2P3efgcOperationsa, bb, fc, fe, fe, ga, ca, eConcurrent?NYYYNYNPhysical time ↓
9. Causal Consistency: QuizValid under causal consistencyWhy? W(x)b and W(x)c are concurrentSo all processes don’t (need to) see them in same orderP3 and P4 read the values ‘a’ and ‘b’ in order as potentially causally related. No ‘causality’ for ‘c’.
10. Sequential Consistency: QuizInvalid under sequential consistencyWhy? P3 and P4 see b and c in different orderBut fine for causal consistencyB and C are not causually related
11. Causal ConsistencyxA: Violation: W(x)b happens after W(x)aB: Correct. P2 doesn’t read value of a before W
12. Causal consistency within replicated systems12
13. Linearizability / sequential: Eager replicationTrades off low-latency for consistency13Implications of laziness on consistencyaddjmpmovshlLogConsensusModuleStateMachineaddjmpmovshlLogConsensusModuleStateMachineaddjmpmovshlLogConsensusModuleStateMachineshl
14. Causal consistency: Lazy replicationTrades off consistency for low-latencyMaintain local ordering when replicatingOperations may be lost if failure before replication14Implications of laziness on consistencyaddjmpmovshlLogStateMachineaddjmpmovshlLogStateMachineaddjmpmovshlLogStateMachineshl
15. Consistency HierarchyLinearizabilitySequential ConsistencyCausal+ ConsistencyEventual Consistencye.g., RAFTe.g., Bayoue.g., DynamoCAPPRAM 1988(Princeton)Part 1: More on consistencyP2: Scalable Causal Consistency
16. Consistency vs ScalabilitySystemConsistencyScalable?DynamoEventualYesBayouCausalNoPaxos/RAFTLinearizableNoScalability: Adding more machines allows more data to be stored and more operations to be handled!It’s time to think about scability!
17. Consistency vs ScalabilitySystemConsistencyScalable?DynamoEventualYesBayouCausalNoCOPSCausalYesPaxos/RAFTLinearizableNoNext Time!Scalability: Adding more machines allows more data to be stored and more operations to be handled!
18. Don't Settle for Eventual:Scalable Causal Consistencyfor [Geo-Replicated] Storagewith COPSW. Lloyd, M. Freedman, M. Kaminsky, D. AndersenSOSP 201118
19. Geo-Replicated Storage:Serve User Requests Quickly
20. Inside the DatacenterWeb TierStorage TierA-FG-LM-RS-ZWeb TierStorage TierA-FG-LM-RS-ZRemote DCReplication
21. AvailabilityLow LatencyPartition ToleranceScalabilityTrade-offsConsistency (Stronger)Partition Tolerancevs.
22. A-ZA-ZA-LM-ZA-LM-ZA-FG-LM-RS-ZA-FG-LM-RS-ZA-CD-FG-JK-LM-OP-ST-VW-ZA-CD-FG-JK-LM-OP-ST-VW-ZScalability through Sharding
23. Causality By Example Remove boss from friends group Post to friends: “Time for a new job!” Friend reads postCausality ( )Same processReads-From (message receipt)TransitivityNew Job!23FriendsBoss
24. Previous Causal SystemsBayou ‘94, TACT ‘00, PRACTI ‘06Log-exchange basedLog is single serialization point Implicitly captures & enforces causal order Loses cross-server causality OR Limits scalability24Local DatacenterRemote DC13241324√ X
25. Scalability Key IdeaCapture causality with explicit dependency metadataEnforce with distributed verificationsDelay exposing replicated writes until all dependencies are satisfied in the datacenter25Local DatacenterRemote DC1324132413after
26. Our ArchitectureA-FG-LM-RS-ZA-FG-LM-RS-ZA-FG-LM-RS-Z26ClientAll Ops Local=Available and Low Latency
27. COPS ArchitectureA-FG-LM-RS-ZA-FG-LM-RS-ZA-FG-LM-RS-Z27Client Library
28. ReadA-FG-LM-RS-ZA-FG-LM-RS-ZA-FG-LM-RS-Z28readClient Libraryread
29. WriteA-FG-LM-RS-ZA-FG-LM-RS-ZA-FG-LM-RS-Z29Client LibrarywriteReplicationwriteafterwrite+orderingmetadatawriteafter =write_after
30. Replicated WriteA-FG-LM-RS-Z30write_after(…,deps)depcheck(L337)deps L 337A 195 dep_check(A195)Exposing values after dep_checks return ensures causalLocator KeyUnique Timestamp
31. Basic Architecture SummaryAll ops local, replicate in backgroundAvailability and low latencyShard data across many nodesScalabilityControl replication with dependenciesCausal consistency31
32. Challenge: Many DependenciesDependencies grow with client lifetimeWriteWriteWriteCurrent WriteReadRead32Same ProcessReads-FromTransitive Closure
33. Nearest DependenciesTransitively capture ordering constraints33
34. Nearest DependenciesTransitively capture ordering constraints34Need extra server-side state to calculate
35. One-Hop DependenciesSmall superset of nearest dependencies35Simple to track:Last writeSubsequent readsThread-of-ExecutionReads-FromTransitive Closure
36. Checking them suffices for causalityCompetitive to eventually-consistent systemNever store dependencies on the serverSimplifies client-side dep trackingClear on every write36Transitive ClosureOne-Hop Dependencies
37. Scalable Causal+A-FG-LM-RS-ZA-FG-LM-RS-ZA-FG-LM-RS-Z37From fully distributed operationwrite_afterwrite_afterreadreadread
38. ScalabilityShard data for scalable storageNew distributed protocol for scalably applying writes across shardsAlso need a new distributed protocol for consistently reading data across shards…38
39. Reads Aren’t Enough39A-FG-LM-RS-ZBossI <3 JobWeb SrvAsynchronous requests + distributed data = ??ProgressProgressProgressTuring’sOperationsNew Job!BossBossI <3 JobBossBossNew Job!New Job!1from 14from 423
40. Read-Only TransactionsConsistent up-to-date view of dataAcross many servers40Logical TimeAlan…Friends111Alan…Status219BossBossNew Job!I <3 JobAlonzo…Friends111AlanAlanMore on transactions next time!
41. COPS Scaling EvaluationMore servers => More operations/sec
42. COPSScalable causal consistencyShard for scalable storageDistributed protocols for coordinating writes and readsEvaluation confirms scalabilityAll operations handled in local datacenterAvailabilityLow latencyWe’re thinking scalably now!Next time: scalable strong consistency42
43. 43