Michael Kaminsky David G Andersen Princeton Intel Labs CMU Dont Settle for Eventual Scalable Causal Consistency for ID: 642242
Download Presentation The PPT/PDF document "Wyatt Lloyd * Michael J. Freedman" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Wyatt Lloyd
* Michael J. Freedman* Michael Kaminsky† David G. Andersen‡*Princeton, †Intel Labs, ‡CMU
Don’t Settle for Eventual: Scalable Causal Consistency for Wide-Area Storage with COPSSlide2
Wide-Area Storage
Stores:Status UpdatesLikesCommentsPhotosFriends ListStores:TweetsFavoritesFollowing List
Stores:Posts+1sCommentsPhotosCirclesSlide3
Wide-Area Storage
Serves Requests QuicklySlide4
Inside the Datacenter
Web TierStorage Tier
A-F
G-L
M-R
S-Z
Web Tier
Storage Tier
A-F
G-L
M-R
S-Z
Remote DC
ReplicationSlide5
Desired Properties: ALPS
AvailabilityLow LatencyPartition ToleranceScalability
“Always On”Slide6
Scalability
Increase capacity and throughput in each datacenterA-Z
A-Z
A-L
M-Z
A-L
M-Z
A-F
G-L
M-R
S-Z
A-F
G-L
M-R
S-Z
A-C
D
-
F
G
-
J
K
-
L
M-
O
P
-
S
T
-
V
W
-Z
A-C
D
-
F
G
-
J
K
-
L
M-
O
P
-
S
T
-
V
W
-ZSlide7
Desired Property: Consistency
Restricts order/timing of operationsStronger consistency:Makes programming easierMakes user experience betterSlide8
Consistency with ALPS
StrongSequential Causal Eventual
Impossible [Brewer00, GilbertLynch02]
Impossible [LiptonSandberg88, AttiyaWelch94]
COPSAmazon LinkedIn Facebook/Apache
Dynamo
Voldemort
CassandraSlide9
System
ALPSConsistencyScatter✖✖
✖✔
✔ StrongWalter
✖
✖
✖
?
PSI +
Txn
COPS
✔
✔
✔
✔
Causal+
Bayou
✔
✔
✔
✖
Causal+
PNUTS
✔
✔
?
✔
Per-Key Seq.
Dynamo
✔
✔
✔
✔
✖
EventualSlide10
Causality By Example
Remove boss from friends group Post to friends: “Time for a new job!” Friend reads post
Causality ( )Thread-of-ExecutionGets-FromTransitivity
New Job!
Friends
BossSlide11
Causality Is Useful
For Programmers:
For Users:
Photo Upload
Add to album
Employment Integrity
Referential Integrity
New Job!
Friends
BossSlide12
Conflicts in Causal
K=2
K=1
K=1K=2
K=1
K=2Slide13
Conflicts in Causal
K=2
K=3
K=2
K=3
K=2
K=3
Causal + Conflict Handling =
Causal+Slide14
Previous Causal+ Systems
Bayou ‘94, TACT ‘00, PRACTI ‘06Log-exchange basedLog is single serialization pointImplicitly captures and enforces causal orderLimits scalability ORNo cross-server causalitySlide15
Scalability Key Idea
Dependency metadata explicitly captures causalityDistributed verifications replace single serializationDelay exposing replicated puts until all dependencies are satisfied in the datacenterSlide16
COPS
Key-Value Store
Causal+
Replication
All
Data
All
Data
All
Data
Client Library
Local DatacenterSlide17
Get
get
get
Client Library
Key-Value Store
Local DatacenterSlide18
Put
Client Library
put
put_after
?
?
Key-Value Store
Replication Q
put
after
K:V
put
+
ordering
metadata
put
after
=
Local DatacenterSlide19
DependenciesDependencies are explicit metadata on values
Library tracks and attaches them to put_aftersSlide20
Dependencies
Dependencies are explicit metadata on valuesLibrary tracks and attaches them to put_aftersput(Key, Val)
put_after(Key,Val,deps)
version
deps
. . .
K
version
(Thread-Of-Execution Rule)
Client 1Slide21
DependenciesDependencies are explicit metadata on values
Library tracks and attaches them to put_aftersdeps. . . KversionL337M195
(Gets-From Rule)
get(K)
get(K)
value,version,
deps
'
value
(Transitivity Rule)
deps
'
L
337
M
195
Client 2Slide22
Causal+ Replication
Key-Value Store
Replication Q
put
after
put_after(
K,V,
deps
)
K:V,
depsSlide23
Causal+ Replication
put_after(
K,V,
deps)
dep_check(L
337
)
K:V,
deps
deps
L
337
M
195
dep_check(M
195
)
Exposing values after
dep_checks
return ensures causal+Slide24
Basic COPS Summary
Serve operations locally, replicate in background“Always On”Partition keyspace onto many nodesScalabilityControl replication with dependenciesCausal+ ConsistencySlide25
Remote
Datacenter
Boss
Portugal!
Gets Aren’t Enough
Remote
Progress
Remote
Progress
Remote
Progress
My
Operations
New Job!
Boss
Boss
Portugal!
Boss
Boss
New Job!
New Job!
You’re Fired!!Slide26
Remote
Datacenter
Boss
Portugal!
Gets Aren’t Enough
Boss
Portugal!
Boss
Boss
New Job!
New Job!
You’re Fired!!
Portugal!
Remote
Progress
Remote
Progress
Remote
Progress
My
Operations
New Job!
Boss
Boss
New Job!
Portugal!
Boss
BossSlide27
Get Transactions
Provide consistent view of multiple keysSnapshot of visible valuesKeys can be spread across many serversTakes at most 2 parallel rounds of getsNo locks, no blockingLow LatencySlide28
Remote
Datacenter
Boss
Portugal!
Get Transactions
Remote
Progress
Remote
Progress
Remote
Progress
My
Operations
New Job!
Boss
Boss
Portugal!
Boss
Portugal!
Boss
New Job!
Portugal!
Remote
Progress
Remote
Progress
Boss
New Job!
Portugal!
Boss
Boss
Boss
Portugal!
Portugal!
Boss
Portugal!
Boss
New Job!
Boss
Could Get
Never
Boss
New Job!Slide29
System So Far
ALPS and Causal+, but …Proliferation of dependencies reduces efficiencyResults in lots of metadataRequires lots of verificationWe need to reduce metadata and dep_checksNearest dependenciesDependency garbage collectionSlide30
Many DependenciesDependencies grow with client lifetime
Put
Put
Put
Put
Get
GetSlide31
Nearest Dependencies
Transitively capture all ordering constraints Slide32
The Nearest Are Few
Transitively capture all ordering constraints Slide33
The Nearest Are Few
Only check nearest when replicatingCOPS only tracks nearestCOPS-GT tracks non-nearest for transactionsDependency garbage collection tames metadata in COPS-GTSlide34
Extended COPS Summary
Get transactionsProvide consistent view of multiple keysNearest DependenciesReduce number of dep_checksReduce metadata in COPSSlide35
Evaluation QuestionsOverhead of get
transactions?Compare to previous causal+ systems?Scale?Slide36
Experimental Setup
COPS
Remote DC
COPS Servers
Clients
Local Datacenter
N
N
N
ReplicationSlide37
COPS & COPS-GTCompetitive for Expected Workloads
High per-client write rates result in 1000s of dependencies
Low per-client
write rates expected
People tweeting 1000 times/sec
People tweeting 1 time/sec
All Put Workload – 4 Servers / DatacenterSlide38
COPS & COPS-GTCompetitive for Expected Workloads
Varied Workloads – 4 Servers / Datacenter
Pathological
Expected
WorkloadSlide39
COPS Low Overhead vs. LOG
COPS – dependencies ≈ LOG1 server per datacenter only
COPS and LOG achieve very similar throughput
Nearest dependencies mean very little metadataIn this case dep_checks are function calls Slide40
COPS Scales OutSlide41
Conclusion
Novel PropertiesFirst ALPS and causal+ consistent system in COPSLock free, low latency get transactions in COPS-GTNovel techniquesExplicit dependency tracking and verification with decentralized replicationOptimizations to reduce metadata and checksCOPS achieves high throughput and scales out