Neil Conway UC Berkeley Joint work with Peter Alvaro Peter Bailis David Maier Bill Marczak Joe Hellerstein Sriram Srinivasan Basho Chats 004 June 27 2012 Programming Distributed Programming ID: 505039
Download Presentation The PPT/PDF document "Logic and Lattices for Distributed Progr..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Logic and Lattices for Distributed Programming
Neil Conway
UC Berkeley
Joint work with:
Peter Alvaro, Peter
Bailis
,
David Maier, Bill Marczak,
Joe Hellerstein,
Sriram
Srinivasan
Basho Chats #004
June 27, 2012Slide2
ProgrammingSlide3
Distributed ProgrammingSlide4
Dealing with Disorder
Introduce order
Paxos
, Zookeeper, Two-Phase Commit, …
“
Strong Consistency”
Tolerate disorder
Correct
behavior in the face of many possible network orders
Typical goal:
replicas converge to same final state“Eventual Consistency”Slide5
Eventual Consistency
Popular
Hard to
programSlide6
H
elp developers
build
reliable programs
on top of
eventual consistencySlide7
This Talk
Theory
CRDTs, Lattices, and CALM
Practice
Programming with Lattices
Case Study: KVSSlide8
Read: {Alice, Bob}
Write: {Alice, Bob, Dave}
Write: {Alice, Bob, Carol}
Students
{
Alice, Bob, Dave}
Students
{Alice, Bob, Carol}
Client
0
Client1
Read: {Alice, Bob}
Students
{Alice, Bob}
How to resolve?
Students
{Alice, Bob}Slide9
Problem
Replicas perceive different event orders
Goal
Same final state at all replicas
Solution
Commutative operations (“merge functions”)Slide10
Students
{
Alice, Bob, Carol, Dave}
Students
{Alice, Bob, Carol, Dave}
Client
0
Client
1
Merge = Set UnionSlide11
Commutative Operations
Used by Dynamo,
Riak, Bayou, etc.Formalized as CRDTs:
C
onvergent and
C
ommutative Replicated D
ata
T
ypes
Shapiro et al., INRIA (2009-2012)Based on join semilatticesCommutative, associative, idempotentPractical libraries: Statebox, KnockboxSlide12
Time
Set
(Union)
Integer
(Max)
Boolean
(Or)
“Growth”:
Larger Sets
“Growth”:
Larger Numbers
“Growth”:
false
trueSlide13
Client
0
Client
1
Students
{Alice, Bob, Carol, Dave}
Students
{Alice, Bob, Carol, Dave}
Teams
{<Alice, Bob>}
Teams
{<Alice, Bob>}
Read: {Alice, Bob, Carol, Dave}
Read: {<
Alice,Bob
>}
Write: {<
Alice,Bob
>, <
Carol,Dave
>}
Teams
{<Alice, Bob>,
<Carol, Dave>
}
Remove: {Dave}
Students
{Alice, Bob, Carol}
Replica Synchronization
Students
{Alice, Bob, Carol}
Teams
{<Alice, Bob>,
<Carol, Dave>
}
Teams
{<Alice, Bob>,
<Carol, Dave>
}
Teams
{<Alice, Bob>,
<Carol, Dave>
}Slide14
Client
0
Client
1
Students
{Alice, Bob, Carol, Dave}
Students
{Alice, Bob, Carol, Dave}
Teams
{<Alice, Bob>}
Read: {Alice, Bob, Carol}
Read: {<
Alice,Bob
>}
Teams
{<Alice, Bob>}
Remove: {Dave}
Students
{Alice, Bob, Carol}
Replica Synchronization
Students
{Alice, Bob, Carol}
Nondeterministic Outcome!
Teams
{<Alice, Bob>}
Teams
{<Alice, Bob>}Slide15
Possible Solution:
Wrap both replicated values
in
a single complex CRDTSlide16
Goal:
Compose
larger application
using
“safe” mappings
between
simple latticesSlide17
Time
Set
(merge = Union)
Integer
(merge = Max)
Boolean
(merge = Or)
size()
>= 5
Monotone function
from set
max
Monotone function
from max
booleanSlide18
Monotonicity in Practice
“The more you know, the more you know”
Never retract
previous outputs
(“mistake-free”)
Typical patterns:
immutable data
accumulate knowledge over time
threshold tests (“if” w/o “else”)Slide19
Monotonicity and Determinism
Agents strictly learn more knowledge over time
Monotone: different learning order, same final outcome
Result:
Program is deterministic!Slide20
A program is
confluent
if it produces the same results regardless of network
nondeterminism
20Slide21
A program is
confluent
if it produces the same results regardless of network
nondeterminism
21Slide22
C
onsistency
A
s
L
ogical
M
onotonicity
CALM Analysis
All
monotone programs are confluentSimple syntactic test for monotonicityResult: Simple static analysis for eventual consistencySlide23
Handling Non-Monotonicity
… is not the focus of this talk
Basic choices:
Nodes agree on an event order using a coordination protocol (e.g.,
Paxos
)
Allow non-deterministic outcomes
If needed, compensate and apologizeSlide24
Putting It Into Practice
What we’d like:
Collection of agents
No shared state
(
message passing)Computation over arbitrary latticesSlide25
Bloom
Organization
Collection of agents
Communication
Message passing
State
Relations (sets)
Computation
Relational rules over sets (Datalog, SQL)Slide26
Bloom
Bloom
L
Organization
Collection of agents
Collection of agents
Communication
Message passing
Message passing
State
Relations (sets)
Lattices
Computation
Relational rules over sets (Datalog, SQL)
Functions
over latticesSlide27
Quorum Vote in
Bloom
L
QUORUM_SIZE
=
5
RESULT_ADDR
= "example.org"class QuorumVote include Bud state do
channel
:
vote_chn
,
[
:
@
addr
,
:
voter_id
]
channel
:
result_chn
,
[
:
@
addr
]
lset
:
votes
lmax
:
vote_cnt
lbool
:
got_quorum
end
bloom
do
votes
<=
vote_chn
{
|
v
|
v
.
voter_id
}
vote_cnt
<=
votes
.
size
got_quorum
<=
vote_cnt
.
gt_eq
(
QUORUM_SIZE
)
result_chn
<~
got_quorum
.
when_true
{
[
RESULT_ADDR
]
}
end
end
Map set
!
max
Map max
!
bool
Threshold test on
bool
Lattice state declarations
27
Communication interfaces
Accumulate votes
into set
Annotated Ruby class
Program state
Program logic
Merge function for set latticeSlide28
Builtin
Lattices
Name
Description
?
a
t
b
Sample
Monotone FunctionslboolThreshold testfalsea ∨ bwhen_true() ! vlmax
Increasing number
1
max(
a,b
)
gt
(n)
!
lbool
+(n)
!
lmax
-(n)
!
lmax
lmin
Decreasing number
−
1
min(
a,b
)
lt
(n)
!
lbool
lset
Set of values
;
a
[
b
intersect(
lset
)
!
lset
product(
lset
)
!
lset
contains?(v)
!
lbool
size()
!
lmax
lpset
Non-negative set
;
a
[
b
sum()
!
lmax
lbag
Multiset
of values;
a
[
b
mult
(v)
!
lmax
+(
lbag
)
!
lbag
lmap
Map from keys to lattice values
empty map
at(v)
!
any-
lat
intersect(
lmap
)
!
lmap
28Slide29
Case StudySlide30
Goal:
Provably eventually consistent
key-value store (KVS)
Assumption:
Map
keys
to
lattice values
(i.e., values do not decrease)
Solution:
Use a map latticeSlide31
Time
Replica 1
Replica 2
Nested lattice valueSlide32
Time
Replica 1
Replica 2
Add new K/V pairSlide33
Time
Replica 1
Replica 2
“Grow” value in extant K/V pairSlide34
Time
Replica 1
Replica 2
Replica SynchronizationSlide35
Goal:
Provably eventually consistent KVS that stores arbitrary values
Solution:
Assign a
version
to each
key-value pair
Each replica stores
increasing
versions, not increasing valuesSlide36
Object Versions in Dynamo/
Riak
Each KV pair has a vector clock version
Given two versions of a KV pair, prefer the one with the strictly greater version
If versions are incomparable, invoke user-defined merge functionSlide37
Vector Clock:
Map from node IDs
logical clocks
Logical Clock:
Increasing counter
Solution:
Use a map lattice
Solution:
Use an increasing-
int latticeSlide38
Version-Value Pairs
Pair = <
fst
,
snd
>
Pair merge(Pair o)
{
if
self.fst > o.fst: self elsif self.fst < o.fst: o else new Pair(self.fst.merge(o.fst), self.snd.merge(o.snd))}Slide39
Time
Replica 1
Replica 2Slide40
Time
Replica 1
Replica 2
Version increase;
NOT value increaseSlide41
Time
Replica 1
Replica 2
R1’s version replaces
R2’s versionSlide42
Time
Replica 1
Replica 2
New version @ R2Slide43
Time
Replica 1
Replica 2
Concurrent writes!Slide44
Time
Replica 1
Replica 2
Merge VC (automatically),
value merge via user’s lattice
(as in Dynamo)Slide45
Lattice Composition in KVSSlide46
Conclusion
Dealing with EC
Many event orders
order-independent (
disorderly
) programs
Lattices
Disorderly
state
Monotone Functions
Disorderly
computation
Monotone Bloom
Lattices + monotone functions for safe distributed programmingSlide47
Questions Welcome
Please try Bloom!
http://www.bloom-lang.org
Or:
gem install budSlide48
Backup SlidesSlide49
Lattices
h
S
,
t
,
?
i
is a
bounded join semi-lattice iff:S is a partially ordered sett is a binary operator (“least upper bound”)For all x,y 2 S, x t y = z where x ·S z, y ·S z, and there is no z’ z 2 S such that z
’ ·S z.
Associative, commutative, and idempotent
?
is the “least” element in
S
(
8
x
2
S
:
?
t
x
=
x)
49
Example: increasing integers
S
=
Z
,
t
=
max
,
?
= -∞Slide50
Monotone Functions
f
: S
T
is a
monotone function iff
8
a
,
b 2 S : a ·S b ) f(a) ·T f(b)50
Example:
size
(Set)
!
Increasing-
Int
size({A, B}) = 2
size({A, B, C}) = 3Slide51
From Datalog
!
Lattices
Datalog (Bloom)
Bloom
L
State
Relations
Lattices
Example Values
[[“red”, 1], [“green”, 2]]set: [“red”, “green”]map: {“red” => 1, “green” => 2}counter: 5condition: falseComputationRules over relations
Functions over lattices
Monotone Computation
Monotone rules
Monotone functions
Program Semantics
Fixpoint
of rules
(stratified semantics)
Fixpoint
of functions
(stratified semantics)
51Slide52
Bloom Operational Model
52Slide53
QUORUM_SIZE
=
5
RESULT_ADDR
=
"
example.org"class QuorumVote include Bud state do
channel :
vote_chn
,
[
:
@
addr
,
:
voter_id
]
channel
:
result_chn
,
[
:
@
addr
]
table
:votes
,
[
:
voter_id
]
scratch
:
cnt
,
[]
=>
[
:
cnt
]
end
bloom
do
votes
<=
vote_chn
{
|
v
|
[
v
.
voter_id
]
}
cnt
<=
votes
.
group
(
nil, count(
:
voter_id
))
result_chn
<~
cnt
{
|
c
|
[
RESULT_ADDR
]
if
c
>=
QUORUM_SIZE
}
end
end
Quorum Vote in Bloom
Communication
Persistent Storage
Transient Storage
Accumulate votes
Send message when quorum reached
Not (set) monotonic!
53Slide54
Current Status
Writeups
Bloom
L
:
UCB Tech Report
Bloom/CALM:
CIDR
’11
,
websiteLatticeRuntime
Available
as a
git
branch
To be merged soon-
ish
Examples, Case Studies
KVS
Shopping carts
Causal
delivery
Under development:
MDCC
, concurrent
editing